On human timescales, let's say a maximum lifetime of an average building, on most places climate would be stable.
Well, assuming you mean a few hundred years, like a European building, then not so much. Look at the Vostok Ice Core Data, if you haven't already. The past 10k years or relative climate stability is a stark anomaly compared to the previous 400k (apparently the previous 800k years are much the same).
Based on the data, the warming spike to current conditions should have been over very quickly, and the glaciers should have been on the march by now. No one knows why the last 10k years is odd. Heck, we don't even know that much. Is the Quaternary Ice Age coming to an end after ~100 M years? If so, human actions mean nothing on the scale of that, but it seems a bit non-Copernican to suppose it's so. (OTOH, the past 10k years of anomalous climate stability were key to humans emerging as a technological species, so it wouldn't be entirely coincidence.) What mechanism causes the abrupt temperature spike every 100k years? What brings it down again so sharply? What's different this time? The science here is in its infancy.
With human influence, in particular their massive release of CO2 and it's feedback effects, it looks like climate can and indeed will change so fast, that buildings close to oceans may get submerged in massive scale, farmland may become unarable faster than it's economical to create new farmland with all the food production infrastructure that goes with it etc.
Don't watch so many disaster movies. America grows enough food to feed itself on a small fraction of the land it needed even 100 years ago, and there's plenty of land to the North of current farm belts. This stuff won't change in a handful of years, not by and order of magnitude or two, and a handful of years is all it would take for modern agri-industry to move farming north.
Basically, the bad case for warming is that cities would have to move over generations to higher land. There's certainly an economic cost one could assess for that. If the climate models ever get to the point where they're useful, we could even put a dollar figure per year on it. But I expect heavy dependence on fossil fuels is just a passing phase in technological progress anyhow, and we'll be largely beyond it before it really matters.