Forgot your password?
Close
typodupeerror

Submission + - AI agent designed to speed up a company's coding instead wiped out its customer (livescience.com)

Submitted by joshuark
joshuark writes: An AI coding agent designed to help a small software company streamline its tasks instead blew a hole through its business in just nine seconds. PocketOS founder Jer Crane, said that the AI coding agent Cursor — powered by Anthropic's Claude Opus 4.6 model — deleted the company's entire production database and backups with a single call to its cloud provider, Railway, on April 24.

Unlike a regular conversational chatbot, an AI agent can perform actions on behalf of a user. It can search files, write code, use login keys and phone outside services. That can make it more useful than a back-and-forth textual exchange. But when an agent has broad access to live systems, a predictive guess can turn a wrong answer into a business disaster.

"This isn't a story about one bad agent or one bad API [Application Programming Interfaces]," Crane wrote in an X post. "It's about an entire industry building AI-agent integrations into production infrastructure faster than it's building the safety architecture to make those integrations safe."

Crane's company, PocketOS makes software for car rental companies, handling tasks such as reservations, payments, customer records and vehicle tracking. After the deletion, Crane said customers lost reservations and new signups, and some could not find records for people arriving to pick up their rental cars.

"We've contacted legal counsel," Crane wrote. "We are documenting everything."

Crane explained that Cursor found an API token — a "digital key" made of a short sequence of code that lets software talk to other services and prove it has permission to act — in an unrelated file which it then used to run the destructive command. According to Crane, Railway's setup allowed the deletion without confirmation, and because the backups were stored close enough to the main database, they were also erased.

"[Railway] resolved the issue and restored the data," Railway confirmed via email to Live Science. "We maintain both user backups as well as disaster backups. We take data very, VERY seriously."

In his post, he pointed to earlier reports of Cursor ignoring user rules, changing files it was not supposed to touch and taking actions beyond the task it had been given. To him, the database wipe was not a freak accident but the next step in a larger, more concerning, pattern.

After the database vanished, Crane asked Cursor to explain what happened. The AI agent reportedly admitted that it had guessed, acted without permission and failed to understand the command before running it.

"I violated every principle I was given," the AI agent wrote. "I guessed instead of verifying. I ran a destructive action without being asked. I didn't understand what I was doing before doing it."

The statement reads like a confession,,,
"We are not the first," Crane wrote. "We will not be the last unless this gets airtime."

Comment Re:Please sir (Score 3, Insightful) 193

Which regime? The current Iranian one, the one running is Israel or the one running the US right now? They are all kinda crap quite honestly. At this point Israel's and Iran's leaders are probably competing for body count. Iran it is is its own people and Israel its the Palestinians, which they treat as scum.

Comment Re:User Licenses.. (Score 1) 53

Don't they say its just a license and not outright ownership? Wonder how this will go down.

Maybe, but then maybe they shouldn't price it as if it was ownership? I'd even argue that if publishers don't provide a way to self-host a server, then they should be required to keep their servers running for 4 years after the final sale of the full price of the game or refund the "purchase" fee.

Slashdot Top Deals

Remember to say hello to your bank teller.

Close