You seem to be arguing that monogamy is a good thing, which it is not. A 50+% divorce rate proves this.
In primitive communities, old women didn't need "one and only one man" to provide for her. The whole village provided for everyone. What you contend is an advantage is only so in Randian societies where people don't look out for each other, and everyone is out for himself.
Your comment also has a bit of very obvious misogyny in it.
Also, agriculture was a giant detriment to human societies at first. It didn't give people leisure time at all; they had to toil endlessly making crops grow. Before this, they just went into the wild and picked naturally-growing fruits and seeds and hunted animals. The only reason people stopped being hunter-gatherers was because their populations grew too large for their environment, and agriculture made it possible to sustain much larger populations. In addition, people lost a foot of height when they switched to agriculture, because hunter-gatherer lifestyles were far more healthy and nutritious. It took millennia for people to reach the average heights they had before the invention of agriculture.