I have lots of thoughts about this. First, go read Resisting AI by Dan McQuillan, it's chock full of why AI in the hands of government is such a problem.
Personally, Anthropic's AI diagnosed a problem during the summer of 2025 that's kept me out of the workforce since 2007- I acquired mast cell activation syndrome thanks to Lyme disease. Now there are a metric crapload of people who've got it due to COVID19, and it's incredibly grim. I've got a relatively mild case - prior to treatment I'd get what looked like a narcolepsy episode in response to eating anything. There are a cluster of mostly women with EDS/POTS and I marvel that they are still alive, because I would not be, given what they face.
I built an app to track my health with Garmin fitness monitor, Apple HealthKit, it'll get reports from Labcorp, and most important of all it's got an internal library. The system will pull stuff from Office of Dietary Supplements, from PubMed Central, and a few other "peer reviewed & meant for doctors" web sites. Being able to constrain what Claude Desktop says by making it use the MCP server has dramatically improved its accuracy.
So ... AI as an assistant is fabulous, but in the government's hands it produces offensive nonsense. I need a benzo if I'm getting in an MRI machine. Oh holy shitkitties, I must be a drug addict, two diazepam months ago have created a continuing train wreck in my care. They issued me Narcan with the little Tramadol prescription I've had for twenty years. I gave it to a couple guys trying to revive one of their friends in the Tenderloin last summer. OH SHIT SHIT SHIT drug addict, the day after I tried to refill it the system started nagging me about my overdue HIV screening.
My doctor gets that I'm a juvenile delinquent with an AARP card, and the only pussy in my life is this obstreperous tuxedo cat that's developed a hobbit like attachment to second breakfast, but the all knowing panopticon? It's delusions would be a lot more amusing if they didn't have a direct impact on my life.