You can address that by having a progressive tax. In the UK, there are tax-free savings accounts that have a limited pay-in amount per year, income on which is exempt from income tax. You could do the same thing with a wealth tax: anything in a tax-free savings account doesn't count. You could perhaps also add an exemption for money in your primary residence, up to the median house price in your region. Beyond that, add a tax-free allowance of something like $50K and most people will pay nothing.
The real problem with such a scheme is that it's open to tax avoidance. It's fine for poor people, whose wealth is typically in cash form and so easily valued, but what about wealth held in private stocks in off-shore corporations? Those currently don't even need to be disclosed, and if they are then it's often very difficult to determine the value of the company (especially if it's a shell company that owns other shell companies that own real assets, with arbitrary levels of indirection in the middle). To make it work, you need complete financial transparency on all private companies.
 When they were introduced, this was about £3K, which was pretty reasonable. If you're earning 50% more than minimum wage in most of the country, you can get close to this. Now it's over £10K, which effectively makes it a tax break for the rich. Unfortunately, it doesn't roll over either, so if you have irregular income then you couldn't put in nothing one year and then £6K the next.