I think if you start to break the problem up into to more manageable parts it starts to become realistic (I know, I know, forest for the trees and all that). One of your points was about universities, but we have already started to build portable and distributed systems of information management (open source search engines like Lucene, or presentation/collaboration systems Media Wiki). So the fact that a university has complex administration and takes up a lot of space on Earth doesn't mean it has to on Mars. We are approaching a point where it might be possible to export and backup all of humanity's knowledge in a device as small as a laptop. Imagine what it could do for the world if while pursuing this larger goal we gained the ability to put a university into the hands of someone with access sun light.
As for the ability to fix or replicate that laptop on Mars, that is a valid point but not necessarily a deal breaker in building a sustainable outpost. Mr. Aldrin is advocating that we start to think bigger, and I think his aspirations for humanity are in line with a more prosperous and exciting future that befits everyone.