I have a site with thousands of products and massive search space due to the number of attributes customers can search on.
The bots are completely stupid trying to product search and iterate over criteria combinations which aren't overall very useful.
You can efficiently grab all our products if a small amount of intelligence is applied and you'd have negligible impact on the server. Instead the bots try to search using each and every possible attribute and therefor blow the cache. I could write a bot to fetch the entire contents of the site in less than an hour and the server wouldn't take a hit; the regular search bots aren't usually pathological about the search criteria.
If the AI ones paid attention to the robots file, and site map, they'd barely be noticed either.
Sure, I can restrict the methods used to search the site, limit it to a cachable subset of search terms, but that would negatively impact users that have unusual requirements.