My guess is because they focused on 'safety'. Late 90's early 2000's, most of the cars were getting ABS, all sorts of airbags, traction control, stupid junk like 'on-star', navigation, power windows/locks/seats/steering columns/etc., Tire Pressue monitoring, climate control... The focus was on luxurious features and safety, which added lots of weight.
And then to add insult to injury the emissions standards got thrown in, and manufacturers were forced to scrap the old way of doing things to meet the new standards. It takes a long time for a new engine design to make it from concept to production, and the manufacturers were sitting on their laurels as it were.