I like Nethack, thank you very much. It's steep learning curve is part of the charm, and the fact that lots of things can kill you. It's not a game for the impatient.
You might want to check out the Diablo series. Blizzard themselves admitted that they were inspired by Nethack (and Angband) when they made Diablo, AFAIK people still play it and is very hyped for Diablo III.
If you want to see popular games with awful UI, you might want to check Dwarf Fortress. Manages to be way worse than Nethack (with even steeper learning curve), but is very popular. Not open source however.
I don't try to claim that the average open source is innovative, rather I was trying to point out a huge exception (innovative dosen't mean it have to be mainstream friendly). I have a feeling it's much easier to get developers together to make a clone of a already loved game (which then can be improved upon), than something new and original.
For innovation, I'd personally rather look amongst what the indie developers offer.