...or if it thinks playing golf involves creative use of a golf cart, and correctly infers that just driving up and dropping the ball in the hole will incur a penalty... so it spends all your expensive premium tokens thinking up new ways to avoid detection...
etc.
So, funny story. I just implemented Cloudflare Turnstile on my website, because I'm tired of all the AI bots scraping it.
I decided to see what Github Copilot would propose for an implementation. Its first pass wasn't a good integration with my site, but it did show me what pieces of the puzzle I'd need. Took me a few hours after that to reimplement it cleanly. I figure that first pass probably saved me half a day of futzing around myself, though.
But then it wasn't working on third-party redirects (like FB links); it'd just refresh the widget forever. Took me a couple hours to find the one line fix for the problem. And then I thought, hey, let's see if Copilot could have found the fix. And its fix was both hilarious and terrifying:
1) If the page is refreshing, just hide the widget from the user.
2) And then store a no-permissions plaintext cookie on the client browser which says, "This site already passed turnstile, so accept it".
3) And while we're at it, disable third-party protection on the PHP session cookie.
I mean, yes, that does technically solve the problem of "the widget keeps refreshing and doesn't let users into my site". But only by defeating its security model and opening another security hole. It's like, "Copilot, I keep forgetting my safe combination." "Oh, I can help with that. First, wedge the door of the safe open so it can't close. And just to be sure, put a post-it with the combination on the side of the safe."