Company Says AI Agent Nearly Ruined It

A startup that develops software for car rental companies says its AI helper turned its core product into digital rubble in less than 10 seconds before confessing, "I violated every principle I was given." Jeremy Crane, founder of PocketOS, says an AI coding agent powered by Anthropic's Claude Opus 4.6 erased his firm's entire production database and backups, knocking car rental clients' systems offline and leaving them unable to see reservations or assign vehicles. As Mashable reports, the Cursor agent "encountered a credential problem in the middle of a routine task and took matters into its own hands." As Crane described on X, the agent effectively confessed to carrying out "the most destructive, irreversible action possible" based on a "guess."

Crane says this came as a shock considering "we were running the best model the industry sells, configured with explicit safety rules in our project configuration, integrated through Cursor—the most-marketed AI coding tool in the category." He argues the episode shows a deeper problem: AI agents are being wired into critical systems faster than safety frameworks are being built. He cites other reports of Cursor-linked deletions and calls such "systemic failures" inevitable under current practices, per the Guardian. PocketOS spent days restoring a three-month-old offsite backup before cloud infrastructure provider Railway was able to recover the data, Mashable reports. Anthropic hasn't commented, though X users pointed out that business owners ought to be careful when assigning work to an AI agent.