> The 'deniers' don't (generally) deny the climate is changing
Correct.
> they just deny it's caused by any sort of human involvement
Wrong. That’s fringe. For the common view, see “denier” Bjorn Lomborg, and, most recently, Bill Gates, who now largely aligns with Bjorn.
> Once the 'breadbaskets' of the world are un-farmable
Yields are strongly overall trending up. The most scientifically supported theory is that “climate change” is slightly slowing the trend as it continues strongly up in absolute terms. Famine has plummeted over the last decades - it’s confined to war torn lawless areas.
> Common logic would therefore suggest to try to minimise the future problem as much as possible
The ‘deniers’ advocate solutions that make a feasible and notable difference, don’t impoverish the third world, aren’t simply feckless Marxist Trojaneering, and don’t give the largest contributors a free pass. Again, see Bjorn Lomborg.