Comment Why not create a standard to deliver to AI (Score 1) 43
I think there could be HUGE benifit, both for AI and Web site owners, if there could be a way of tellling/redirect AI Bots or Search Bots to a data index, incremental even, to crawl. Skipping all the javascript and html bullshit.
Sites could deliver RAW data for bots to digest, with pointers to the "human-web" - that is, disallow scraping, but giving the same or part of information to use for search and AI indexing... Like checking the box "I AM a robot", and then get a robot friendly datastream...paid or free.