It is true, as duly reported by athropologists, like theo ne I cited above. Within the same regions, the primitive farmers are more often victims of famine than their hunter-gatherer cousins living next door.
then why was agriculture even invented?
Because it allowed for planning and higher population density, in particular a much higher number of children.
how did agriculture based communities overtake hunter-gatherers in population, if there was no scarcity of food?
See the previous point. More population growth leads to morel and needed to sustain it, which is why agricultural peoples spread themselves, conquering the lands of hunter-gatherers until they had pushed them all onto least useable land.
Do you think people would spend that much energy if food was plentiful?
Surveys done by anthropologists show that both hunter-gatherers and primitive farmers spend about 20-24 hours of their week working at "producing" food, with a slight advantage in raw number of calories (2140 daily average) and much bigger advantage in terms of food diversity (upwards of 70 different plant and animal sources) going for the hunter-gatherers.
Switching to farming is not a matter of work spent (apart from the food diversity it's pretty much the same), it's a matter of being able to plan for the future and especially plan for feeding more children as well as securing food supplies in case of a catastrophe such as the Great Thawing Up of 15000 B.C.