A home is normally the single biggest item a person spends their income on, and with good reason
For a typical middle-class person, sure? For someone on the lower end of the upper class even, probably (the tax break is tough to beat). But for the true upper class, you should be spreading your investments around. At that level, you have plenty of other useful things you can do for investments and tax dodges. Unless you want to spend all day following the market, probably the best thing to do would be to put most of it in big indexed funds, and live off the interest (aka: a "Trust Fund"). But I'm not in that weight-class, so someone there could probably advise you better.