I've been participating in the NYC BigApps string of hackathons this Spring. They really shouldn't be called "hackathons" because, as the submitter said, they're really just pitch-a-thons. Three weeks ago we showed up to the first, came up with an idea on the fly, banged it out in two days; then, when it came time to present the app we had done every other team stood up and presented apps they had been working on for years.
Naturally, something that has been in development for years is going to be more complete and polished than something that was born 48 hours before. And that long-term project is more likely to win, and win they did. In the subsequent two hackathons we also presented stuff we had been developing for a long time and won both times. But it felt wrong. It felt like it was violating the spirit of what a hackathon should be.
What hackathons should be is a crazy all-night code fest of how quickly techs can move ideas from conception to reality. 48 hours is an absurdly short period of time to create. All of us who develop for a living know that. But that intensifies the design/scope decisions you have to make, the team collaboration you have to effect on the fly, and the exhiliration of a win if you can pull something off.
Finally, the panel of judges should be diverse, cutting across generations and disciplines, because young 20-something techs are perhaps not always the best positioned to see the potential of an app in the bigger societal context.