I've always found "I don't care how I look" to be the geek equivalent of the general public's "I don't understand maths". Both are shouted proudly, to separate oneself from the other group (those vapid socialites who base everything on appearance/those sweaty geeks who spend their lives in the basement with lines of code rather than people), when really neither are attitudes to be proud of. Of course there are times when there's no harm in looking a mess, but that's rarely what people seem to mean - it's often more of a day-to-day lack of care.
Sure, in an ideal world, people wouldn't judge on appearance, but that's not the planet we live on - there's no need to be uncomfortable, or look like a corporate drone, or even stand out particularly if you don't want to, but a high quality pair of jeans (the difference between good and crappy is vastly noticeable, even if you can't put your finger on exactly why that is) paired with a well-fitted button down shirt, a decent belt, and a good pair of shoes takes no more real effort than cargo pants and a t-shirt. Both are socially acceptable, but the former will immediately make a better impression on pretty much everyone you meet - they're better disposed to you, you feel more confident as a result, and so it goes on. The latter, to most people, would be the equivalent of needing to take out your phone to calculate a simple tip while you're out to dinner with a bunch of engineers - again it'd probably pass without comment, but it'd leave a subtle negative impression and modify people's disposition (conscious or otherwise) towards you as a result.