I'd like to see a lawsuite concerning crawling information in robots.txt.
I don't think it would stand up, it's a voluntary, community-created standard. It's a polite way of saying "this is dynamic information or private". If it's private, then don't make it public!
The first things hackers and pentesters do is traverse robots.txt