Yes, or to put in the words of businesses say, it is all about managing your cost structure. Paying that mortgage off buys you a whole bunch of cash flow to work with. You can invest it, spend it (or a combination thereof). Spending it could mean taking a pay cut by getting that less stressful job/career.
Well, you are right, paying off the mortage will lead to higher cash flow in the future, but at the cost of lower cash flow now. If you invest wisely in a diversified portfolio consisting of a healthy mix of index funds, mutual funds, bonds, and some cash in a money market you should easily be able to beat the interest cost through gains, dividends, bond payments, etc. over the long haul. Over 10-20 years the market tends to go up at much more than 6% per year (about the real cost of a morgtage after tax, it's hard to say what it really is though since eveyone gets different rates and has different tax rates), bonds tend pay a steady stream at between 5-10% depending on the bond (Treasury bonds are lower right now but even though interest rates have been higher lately they are still lower than the historical average), and a money market will get you about 4% these days and tends to be indexed below the T-Bill rate.
Paying off principle only helps build equity, but if you buy a house and can get it to increase in value you have used the banks money to build your own equity. It is just like leverage in the business world, using someone elses money 9the banks) to make yourself money. Now all of this is not for the feignt of heart, when you loose money the loss is amplified as well. Right now we are seeing a lot of the down side of this because people got crazy in some markets such as FL and SO CAL and just kept buying houses to flip, developers couldn't stop building, and then the number of people looking to move to these areas declined. The developers have a hard time knowing what real demand is because such a large number of their sales were to flippers so the markets got saturated with too many homes. The "bubble" will end and another will come, it is the nature of a market with imperfect information.
If you are close to retirement the situation may change, but this depends on the individual financial picture.