Yields don't have to be much lower. Growing organic is more expensive because it requires more labor hours, which is the big reason industrial farms were so easily honeymooned by Monsanto and other agri-businesses to use the latest pesticides with minimally studied safety profiles, fertilizers, and genetically shotgun spliced crops that cost a fortune to the farmer and lock them in to business with those companies. It also allowed farmers to forget the hard won knowledge over generations of how to maintain their soil and do proper crop rotation to prevent degradation of their land.
My wife grew up in rural Illinois. She remembers having to come in when the spray planes would fly overhead, and the stench that lingered in the air afterward. She got a non-cancerous brain tumor that will probably reoccur over her life. She has had lots of friends from the area live with and die from various cancers before they've turned 40. There is no hard scientific evidence this spraying was the cause. But those chemicals are generally studied in isolation, in labs for several years, not together in the environment with humans over decades. She has some concern that we may learn many decades later there is a link.
And as one other poster alluded to, if you are concerned about calorie yield per acre, you'd do much better by not (or rarely) eating beef as it is very land intensive to grow the feed for cows and they have a high global warming contribution through their methane emissions.
I'm aware of the potential for observation bias and anecdotes aren't scientific. I'm also aware of the low threshold for efficacy even for approved drugs, and while a study may not find statistically significant correlation on the whole, that doesn't mean some individuals weren't greatly benefited. In the end if you feel something is really beneficial and the risks are low, you just got to Let the science be damned . Sometimes the science catches up with you.