You're really underestimating the social part of face-to-face meetings.
Those moments discussing other things, laughing about irrelevant jokes, etc are not lost time. These moments are valuable as well, they help people get along with one another which in turn helps the whole process. It helps defuse arguments, it helps people to be nice to one another and to cooperate. People are, by nature, social creatures. You may figure you're an exception, it may indeed be true that not everyone can exchange information in person, but I'm sure the other 99.999% or so of people in this world do so just fine, some arguably better than others but overall people can manage to do this just fine.
Face to face meetings, especially those late at night in the bar over a beer or two, are also the place that people feel free to launch the craziest ideas. Ideas that pop up, are on second thought totally stupid and impossible, but may just spark another idea that gets the group closer to the solution - or at the very least you can have a good laugh about, which again is good for the group as a whole. This are things that are rarely done on e-mail as everything there is on record, and definitely not everything that's being said in a meeting has to stay on record. Only the final conclusions with possibly some arguments on why that decision was made.
For the other extreme - groups trying to work together without any face-to-face meetings, be it formal or informal - just see what happens on Internet mailing lists. That often degrades into a flame war, people fighting with one another over the most irrelevant details. This may take even more time than the off-topic time in face-to-face meetings. The main reason it seems to work is that all participants are so driven to get the project done.
Both have their place, and I think the face-to-face meeting should come first (mainly for the social part), e-mail second. The meeting is great for discussing what's to be done, e-mail/IM is great to follow up on specific details - either one to one or in small groups.