Comment Re:tell me about it (Score 2) 45
>"I run a very small boutique hosting service and traffic has more than doubled since AI, all attributable to them. OpenAI in particular just seems to come along and hit like 30-60K links per day, no robots.txt rate limiting, just a "gimme all your data" scraping posture. Amazon is by far the worst, "
Oh, there is far worse now. Last week my club website was hit by a full-on distributed bot-net scraping our wiki. Up to a dozen hits per second from 250,000 unique IP addresses all over the world (but mostly Singapore and Brazil addresses). All of them ignoring robots.txt. All of them pretending to be people/browsers. It crippled the old/small server, so I took the content down and the traffic continued for 5 MORE DAYS still requesting various links into the content, until it finally started tricking to 2 or so per minute.