This happened to me the other day too. I don't know why I get suckered into it.
I had a problem where my LSP was returning ??? at the beginning of completions; there were some non-UTF-8 characters in there and I didn't know where they came from. ChatGPT said it was a known problem from some Visual Studio compile flags when they get passed to Clang. In retrospect, that's stupid.
But it said it could solve the problem for me by intercepting the .rsp files, stripping the bad flags, and then passing them on. It wrote a little python script to launch the lsp and strip the flags.
I ran it and it seemed to work, but my testing protocol was wrong, and I was just seeing some correlation--the problem was intermittent, and me putting the script in place happened to coincide with some good responses from the LSP.
When I tried shutting down emacs and starting it again, it couldn't stay connected to the LSP through the script, and of course it couldn't. So the script was never really in play. After searching around for a while, it was a known error with a package called eglot-booster, and it was easy to fix by turning off a particular flag.
I have since instituted a rule where it MUST, WITHOUT FAIL, provide a citation for anything it tells me. If it can't find a citation, it MUST mark the section as unverifiable. That has helped a lot.
From my experimenting, Gemini is excellent at giving citations, but it gives up very easily if you tell it to do something and it can't do a complete job. I tried to have it summarize all the commands in a package that started with : (colon). It did an incomplete job, and when I asked it to do it again, it just told me where the documentation was. Claude did an excellent job. ChatGPT also did a partial job, but maintains conversational context better, so you can often get it to fix its own problems.