I have either read or given up on all of the nominees. I am not convinced that Redshirts was the best novel, but it was probably the intersection of "mainstream / well-known" and "not so bad." Remember, Hugos are determined by a vote of science fiction fans at the convention (or who bought "supporting memberships"), and there's no requirement that they read all of the books.
The nominees were (in order of placement)
5. Blackout - the third in Mira Grant's Newsflesh zombie series. These books are entertaining, and the setting is fairly clever - where zombies are just a fact of life, and an ongoing danger - but this wasn't the best of the trilogy, and the previous two were nominated but didn't win.
4. Throne of the Crescent Moon - an entertaining fantasy novel with a lot of Arabic mythology as an influence. I enjoyed it thoroughly, but it was the author's first novel, and it's not as well-known as the others on the list.
3. 2312 - Another Kim Stanley Robinson book - a little heavy on geology, and a little meandering for my taste, but interesting in bits. KSR has won Hugos before, and is pretty well-known.
2. Captain Vorpatil's Alliance - part of the whole Vorkosigan / Miles Saga from Lois McMaster Bujold. I usually like her books, but something about the voice this one was in just bothered the crap out of me, so I didn't finish it. Once again, Bujold has a lot of past Hugos, and this is like book 15 in the series, so it definitely has a constituency.
1. Redshirts - has Scalzi's usual flaws with fairly bland characters and a little bit of generic plotting. But it was funny, and ended up being at least a little touching and thought-provoking.
Essentially, the fan vote gives a strong advantage to well-known authors. Of the books that apparently had a chance (the top 3), Redshirts might have been the best. (Full stats at http://www.lonestarcon3.org/)