Isn't population growing mainly due to latency?
Not quite.
In the developed world, population growth is negative absent immigration. Currently, this applies to China, the EU, and the USA. Last I bothered to check, the projections were for continued global population growth up to the 10-15 billion range, followed be a decline to a stable population in the 5-9 billion range.
Note that that "stable population" presupposes that the entire world is "developed" by that time.
If owning robot overlords can assure you all you ever need without working, it's obvious everybody will want these, but only the most fortunate will afford it, leaving the rest of us in misery.
If we don't have robots making our shit, then we'll still be making it the old-fashioned way, which sort of implies we'll be working for a living.
At least until we can set up robot factories of our own, of course.
Look at it this way - if the robots can make enough stuff for everyone to have everything they want, then everyone will be "rich". Of course, the real problem in that case is that we'll stop measuring "wealth" in "things we own"....
Also note that even if the (currently) wealthy types make just enough stuff for themselves and leave the rest of us to rot, there's nothing actually stopping the rest of us from building our own robots to make shit for us....