First of all, everything should be archived for future generations and researchers. Otherwise, it defeats the whole point of the project.
But for the general public, the robots.txt should be honored and content hidden with a few conditions. First of all, it should not be retro active. I've seen valuable information lost when domains have changed name and the new owner has blocked the contents with a robots.txt. Second of all, there should be a review system to override the robots.txt. For example, if a site is cited in Wikipeida, it the robots.txt should be ignored and hidden content unblocked.