Let's follow your logic.
If a home breaks (or gets submerged) we'll build another one. Okay - how much is Miami worth? Or New Orleans? Let's say we need 1 million new homes - at $150k each that will be $150 billion. Now we also need the infrastructure - roads, schools, power lines, gas lines, highways, etc. Let's call that another $150 billion. Now we'll need the factories, buildings, strip malls and all the other places for these people to work. Call it another $150 billion. So, to ballpark it, we need $500 billion to create a new city of 4-5 million people. (4 million is the estimate for number of people displaced from rising sea levels.)
As for farming, you really don't understand the specifics of food production. Plants need certain amounts of heat and sunlight. Unfortunately, as places further north warm up, it doesn't mean we can suddenly grow "southern" crops - the length of day isn't right. Sure - they will probably grow, but the yield will be much lower for a decade or two until seed companies can hammer out the right genetics. (If they can.) Which means - lower food production. This also ignores the differences in soil and everything else that have established where we grow certain plants.
It's true there are other things that make up society - and those won't go away. Unfortunately, no one is going to want to share their rich, fertile land with people who were displaced by rising sea levels which means eminent domain claims will be tied up in the courts for years. Plus, most towns don't like large numbers of "outsiders", so look for increased enforcement of laws regarding homeless people. No one is out to get the homeless - until they are in your neighborhood.
Oh - to answer your original question: Where is the proof that infrastructure is brittle? Take a look at rolling blackouts, gridlock during rush hour, failing railroad lines and the volatility of gas prices. That's your proof. With good infrastructure those would be eliminated. (Gas prices spike every year when refineries have to perform maintenance or switch what they produce.)
The real problem with your argument can be traced to the basic principle: In theory, theory and practice are the same thing. In practice - they aren't. In this case - in theory we could just rebuild everything but, in practice, there are a million things that will go wrong.