I believe you, but only because of the particular details you included: you train hard every day, and you used to be overweight and sickly. Otherwise, even a reasonably fit 50-year-old is no match for a normal, unathletic 20-year-old.
At 40 I was slim, fit, and healthy, with good stamina, and sexually very active. At 50 I was beginning to notice changes, and my activity level dropped. By 60, all the above had changed dramatically, due in part to specific medical issues.