Submission + - AI agent designed to speed up a company's coding instead wiped out its customer (livescience.com)
Unlike a regular conversational chatbot, an AI agent can perform actions on behalf of a user. It can search files, write code, use login keys and phone outside services. That can make it more useful than a back-and-forth textual exchange. But when an agent has broad access to live systems, a predictive guess can turn a wrong answer into a business disaster.
"This isn't a story about one bad agent or one bad API [Application Programming Interfaces]," Crane wrote in an X post. "It's about an entire industry building AI-agent integrations into production infrastructure faster than it's building the safety architecture to make those integrations safe."
Crane's company, PocketOS makes software for car rental companies, handling tasks such as reservations, payments, customer records and vehicle tracking. After the deletion, Crane said customers lost reservations and new signups, and some could not find records for people arriving to pick up their rental cars.
"We've contacted legal counsel," Crane wrote. "We are documenting everything."
Crane explained that Cursor found an API token — a "digital key" made of a short sequence of code that lets software talk to other services and prove it has permission to act — in an unrelated file which it then used to run the destructive command. According to Crane, Railway's setup allowed the deletion without confirmation, and because the backups were stored close enough to the main database, they were also erased.
"[Railway] resolved the issue and restored the data," Railway confirmed via email to Live Science. "We maintain both user backups as well as disaster backups. We take data very, VERY seriously."
In his post, he pointed to earlier reports of Cursor ignoring user rules, changing files it was not supposed to touch and taking actions beyond the task it had been given. To him, the database wipe was not a freak accident but the next step in a larger, more concerning, pattern.
After the database vanished, Crane asked Cursor to explain what happened. The AI agent reportedly admitted that it had guessed, acted without permission and failed to understand the command before running it.
"I violated every principle I was given," the AI agent wrote. "I guessed instead of verifying. I ran a destructive action without being asked. I didn't understand what I was doing before doing it."
The statement reads like a confession,,,
"We are not the first," Crane wrote. "We will not be the last unless this gets airtime."