Your error is in assuming a simple, isolated system and ignoring the complexity of dealing with the horribly analog world of biology.
In general, there are two considerations for when, and how much, plants grow. The first is the amount of sunlight they receive (hours per day) and the second is the number of "degree days". Since duration of sunlight isn't going to change (at a certain latitude), let's focus on "degree days" first.
A "degree day" is based on the temperature of the day, so the higher the temperature - the higher the value. However, there are bounds for this. For example, corn needs at least 50 degrees Fahrenheit, but not more than 86 degrees Fahrenheit. i.e. - Below 50 means "0 degree days" and 92 will be the same number of degree days as if it were 86.
The problem comes in when it is far too warm which, for corn, comes in around 86 degrees. The plant hasn't adapted for growing in temperatures much higher, and will shut down growth; much higher temperatures will even cause damage to the plant. Here is a human analogy - a human might be able to run really fast and really far but, if it is 115 degrees outside, that isn't going to happen and any activity may result in heat stroke. A plant will be stressed in this kind of heat and will actually be damaged. In this way, too much heat will cause plants to grow less, and we will have lower yields.
However, since plants also depend on certain amount of sunlight, it isn't a simple matter of moving things northward (or southward in the Southern Hemisphere) to match temperature. All of the plants are also expecting a certain duration of sunlight. This isn't constant with latitude, so moving the plants north will reduce yield. (And more sunlight doesn't mean higher yield - plants also do things at night like release water vapor.) This means that we will have to reengineer our crops to match new conditions - which will take decades. (And crop genetics isn't a simple matter - companies spend billions on trying to make better species.) So, until we do that, we will have lower yields.
Also, many plant diseases like the heat (or like that they don't freeze to death in the winter - see Asian Soybean Rust ranges) - so they will enjoy millions of square miles of new territory - increasing the cost of production (herbicides and pesticides) and, since bugs and molds eat the plants, will give us lower yields.
The other problem is related to economics and infrastructure. Farmers have certain equipment to plant and harvest the crops native to their area. Plus, their fields have been designed for those certain crops. For example, they may be terraced in a certain way or be designed with a certain level of drainage based on existing weather patterns (temperature and rainfall). Renovating millions of square miles of farmland is going to be expensive and ridiculously time consuming and until it is modified to match new, prevailing weather patterns, will contribute to lower yields.
The other side to the economic coin is that decisions are not going to be made on a 50-100 year strategy. To operate next year, a farm needs to turn a profit this year. So, they aren't going to completely retool if yields go down 10% - it would make no sense. The capital costs would dwarf any profit from the new crops being put in. Therefore, they will operate at lower capacity and accept a lower profit - since it is still a profit. Sure, we will get changes when push comes to shove, but that will take decades as climate change forces them to change. Until that point - lower yield.
Moral of the story, we are looking at decades of lower yields as climate change really kicks in.
