Itâ(TM)s in its name. SERP = Search Engine Results Page. Theyâ(TM)re scrapes of Google SERPs. This is fallout from the Reddit license where Reddit decided to collect rent from Google and others for scraping their content. Apparently Reddit and Google decided to put Google only visible stuff on their pages (which is explicitly illegal under Googleâ(TM)s TOS, and has resulted in index banning) and then served up this secret content via SerpAPI.
Scraping Google SERPs has been standard behavior for literally as long as Google has existed. Thatâ(TM)s literally how Facebook, Microsoft, and countless startups and academics evaluate their own search engines. Iâ(TM)m not exaggerating. They literally compare their results to Google results, which always made me wonder what Google does.
As far as ignoring robots.txt and using different IPs? Please. Thatâ(TM)s also has been standard behavior for as long as the web has been around.
This is monopoly behavior, and Google is openly engaging in it and attacking the open web because thereâ(TM)s a sympathetic White House administration for them.