The explanation is simple. Today we have a lot more competition than 40 years ago. Some of it was in part caused by de-regulation. Other part was caused by the innovative "low-cost" carriers that put real pressure on traditional airlines. Finally, the internet increased the completion even more. Today finding the lowest priced fare is a click away. As a result, all airlines compete strictly on price while saddling the customers with hidden fees, overbooked flights, flight delays, and less comfortable cabins.
And despite all the whining, this formula works for consumers too. No one wants to pay $700 of 1999 dollars for a flight that costs $400 today, even if it means putting up with small legroom, nickel and dimming, no in flight food, and TSA harassment.