@yolodude you did not mention how OLD you are... that has a lot to do with it. A teenager can get hard in an instant, even when they don't WANT to. I can't say for sure, but I think by age 80, getting an erection is next to impossible without pills and even implants.
Cock sizes also seems to be a factor. Guys with huge cocks have a difficult time getting hard. A good person to ask about that is RaphJD. Ask him how long it takes him to get his gigantic cock hard. Probably not long, since I think RaphJD is only about 30 years old.