Also, people during that age were not especially healthy. They probably died in their 40s.
Wrong. Half of them died young (typically before the age of 5) and the rest lived to their 60s and 70s, sometimes even older. Reconstructed modal age for primitive hunter-gatherers is 62 to 64 years of age.
There is a marked reduction in average size, and sudden appearance of generalized tooth decay, traces from infectious diseases and formerly absent bone deformities in our record of skeletons from the paleolithic to neolithic transition. Granted, the infectious disease became more widespread because of the growing densities of populations at the time, but the rest has been determined to come from the evolution of the diet. There is also a reduction of serious injuries observed, because less hunting decreased the exposition to dangerous predators and hunting accidents.
As for life expectancy, it decreased slightly with the agricultural revolution until circa 2000 BC, at which point advances in hygiene, sanitation, productivity and trade compensated for the difference. And we only now have caught up the loss in average height. There has been evidence of an adaptation to agricultural diets over time, but its effect is still small in terms of life expectancy.