BOTH screwed up. The system shouldn't have made the match and the human should have done a better job seeing that the match was wrong. I'll go one larger and say the people who created the procedures screwed up because they didn't make allowances for the possibility of the first two screw-ups.

The problem is, people have been over-trusting technology since technology was invented. Consider the '70s when corporate representatives were so fond on the phrase "But the computer says...". Yes, you could hear the bold italics in their voice, as if it was the infallible final word on everything.

If THE COMPUTER says you shoplifted, rest assured, some dolt in a suit will treat that like God's own word booming from the clouds.

But every person who sets foot in that store probably "kinda" looks like someone who has shoplifted. And since shoplifters look like everybody else, everyone COULD be a shoplifter.

Many have voted with hairspray and a sawzall.

It's worse, they don't have to go sentient at all. The black death wasn't sentient but it wiped out whole towns anyway.

Rogue agents with no awareness have already demonstrated that they will act against any entity with a conflicting goal, including humans.

IIRC, more than one lawyer has gotten in to hot water because of AI hallucinations in legal briefs. They sign their name to those, they are responsible.

The legal process is adversarial but it is not anything goes. Fabricating evidence and perjury are out of bounds for example.

I think this falls into that category.

OTOH, if the court DID use AI to make decisions, the whole court would invalidate itself, so there is some ethical wiggle room.

AI should have no agency at all. It should only advise the human in the loop who is actually responsible for what happens.

Actually, without a need to pay for servers (virtual or other), it offers freedom from sponsors that want the conversation kept profitable and ad friendly.

To me it sounds like a newb wrote a web app and didn't quite know what an API is.

Did it use the API normally or did it do an SQL injection?

It's still a cloud, just one behind a military firewall.

And the AI circumvented them because of bad coding. That part of the story is not at all new.

Or more to the point, why on the open internet? Put it on 10/8 or unallocated IPv6 somewhere and VPN it through the public internet if they must.

That way, if the camera tries to phone home, it won't find a router willing to complete that call.

Banks are supposed to lend money. I just saved them the paperwork.