Your general conclusion is correct, but your analysis is wrong.
Wealth isn't a simple concept. It has no direct relationship to anything physical, but it's got a LOT of indirect relationships.
E.g. part of wealth is being healthy. This depends on physical things, but for most of them there are tradeoffs. Potatoes aren't necessary if you've got wheat or oats. Clean air is valuable, but so is treatment for diseases. And NONE of these things are directly proportional to money.
E.g., part of wealth is having a happy marriage. Lack of poverty sure helps that, but anything beyond that doesn't help much.
The wealth of a country can increase when it stops being "top dog", but this is for sure by no means guaranteed.