10 million is sufficient to assure genetic diversity.
******
As a point of comparison, assume everyone dies at age 80, starts working at 20, retires at 65. That's 43.75% of the population not working, or by your figuring 1.688 people working to support 3 people. Initially, that sounds like an excessive burden, but it requires a closer look.
Median household income in the U.S. is $84,000. FWIW, that includes single parent homes, and I don't know how to factor that into calculations.
In a non-nanny-state society, everybody is responsible for himself and the results of his actions. His responsibilities include supporting his children until they can support themselves. That need not be an overwhelming cost if it were not for the excessive costs of education. (For 2 children in the U.S., kindergarten + 12 years grade school + 2 years college at $20,000/year is $600,000. {Thanks, teacher's unions.} 20 years food for 2 children $100,000. Incremental rent for a larger apartment or the incremental cost for a larger house, $100,000. Other costs are usually small in comparison. That's $900,000, 2/3 of which is government-mandated education. In a free market a better education for 2 children could be provided for about $120,000, bringing the total to $320,000.)
From 65 to 80, a person should be living off what he accumulated during his working years, (In the U.S. 45 * $84,000 = $3,780,000 gross, minus expenses.) That is not living off someone else's production, that is living off the surplus created during a lifetime. A person is not being supported by someone else when he's sleeping or on holidays or weekends. All that changes in retirement is the timescale.
Taxes and support provided by government do not change the financial principles involved, that should be "money in" = "equivalent to money out." But government is inefficient, and that's a problem.
In summary: A lifetime income of $3,780,000 to support rearing 2 children for $320,000 to $900,000 leaves $2,880,000 to $3,460,000 for a couple to live from age 20 to 80. This should not be a problem or a reason not to have children. Arguing that it takes X people to keep Y people alive is pointless. Over the course of history, production becomes more efficient due to technology, meaning that it will take even fewer people to support 3 people at the same level of well-being, and that's a good thing.