You're all missing the point, this guy has no ethics... "I used robots.txt for info", care to think about that statement.

Speaking of one of the grand-daddys of reusable source, does anyone remember the DECUS Tapes from the DEC community. There's still a lot of prior art work to be found there.