Sociological studies of hunter-gatherer societies have indicated that they even now have more free time than we do, not less. Moreover, it was only within the last 400-500 years that agricultural societies began to overtake hunter-gatherers in terms of nutrition (as measured by looking at the height of skeletons, and signs of the presence of malnutrition-related diseases). In other words, it was only very recently that agricultural civilization became good not just for those at the top but also for the majority.
The argument, then, for why agricultural civilization came to dominate the world even if it did not result in a better quality of life is this: Although the diet of cheap carbohydrates provided by agriculture did not result in healthy people, it did provide energy to sustain more people (albeit with a lower quality of life), whereas hunter-gatherer civilizations need to practice contraception and infanticide (and they did, and do, both) to avoid overexploiting their range. The societies with larger populations (the agricultural ones) were, in turn, able to field armies and otherwise exert power in ways that hunter-gatherers were not, and in this way also out-competed them.
In other words, until very recently, if you wanted to create a large and powerful society at the expense of individual health and leisure time, your best bet was to practice agriculture. If you wanted to create a small society of well-nourished and healthy people with more leisure time at the expense of collective power, you'd want to pick the hunter-gatherer lifestyle. And even now, although hunter-gatherers no longer have the nutritional advantage, they still win on leisure time.