We are fully grown by our late teens/early 20's. We are at our physical peak around that time. We have the energy for child rearing. It would make sense to bear and raise children in that neighborhood. The extended childhood experiment that we tried out some time ago worked pretty well, it allowed for teenages to gather a little bit of wisdom before getting out on their own. But there are limits to extended childhood, and the extreme narrowing of fertility.
Yes biologically speaking this makes sense, but our species also wants to be happy and breeding tends to make us unhappy and want to die soon. Children suck all the life out of you like little vampires and makes the mother less attractive (breasts like half deflated water balloons and mega-vagina) and sometimes less interested in sex. In some species the mother dies soon after giving birth. I think it is true in our species as well except that the death is a psychological one: the death of all enjoyment of life, of all happiness and it applies to both parents and not just the mother. So if you want to live a happy life it makes sense to have children only after most of the enjoyment of life has already passed. So I think waiting for the last few eggs to have children is a great idea. Yes it's more selfish, but we only get one life. Why not surf that wave right to the beach?
For men I think the ideal time to have kids is in your 50s or even 60s. For women early 40s is optimal if you are going for the one child ideal. 38 or 39 if you want 2 or 3 of those life-stealing, mole-rat looking creatures. I'm not yet 50 and have never had any desire to have children but it is too early now anyway. Child rearing is something you do when you are getting ready to die.