Of course the driving force behind building up our civilization has come from the more temperate zones of our planet. People living in the "horrible" environments of the world are mostly subsistence survivors living off the land.
You may not be worried about a few feet of sea level rise or a degree or two of warming but what makes you think that's the end of it? What do you think your children (or your relatives children) will have to contend with after you're gone? Current worst case scenarios show around 6 feet of sea level rise and 4-6 degrees C of temperature rise by 2100.