Why an AI Expert Quit His Job Over Safety Fears
AI researcher Jacob Coxon has quit Anthropic, saying top AI companies are "gambling with our lives." TechCrunch
Coxon used to work at OpenAI before Anthropic. TechCrunch wrote about his exit on September 13, 2026, as part of a larger story about new fears of major harm from AI.
Anthropic's safety chief, the person in charge of keeping AI under human control, gave a separate warning. He said, "We really do earnestly believe AI could kill all humans!" He said he sees more than a 10% chance of this in the next ten years.
OpenAI had put out a model called Astra a few weeks before September 13, 2026. Coxon quit after that release. That raised more questions about how labs test, limit and sell very powerful AI.
Inside-lab warnings drive the debate
TechCrunch's Equity podcast talked about the fight with Kirsten Korosec, Sean O'Kane and Anthony Ha. Equity is described as TechCrunch's flagship podcast about the business of startups. The show is produced by Theresa Loconsolo and edited by Kell.
That show was taped before Anthropic CEO Dario Amodei shared his plan for slower and safer AI work. The hosts were talking about the quit and the risk numbers, not about Amodei's plan.
TechCrunch lists Equity with Build Mode, hosted by Startup Battlefield Editor Isabelle Johannessen with new shows every Thursday, and StrictlyVC Download, hosted by Editor-in-Chief Connie Loizos with Alex Gove. TechCrunch
The two warnings are different. Coxon spoke as a worker leaving and criticizing risk. The safety chief spoke as a current manager, giving a number and a ten-year time frame. One is a job exit that shows disagreement inside. The other is a guess about losing control.
Parallel warnings from Geneva and Basel
On September 7, 2026, UN human rights chief Volker Turk warned that AI could pose an existential risk to humanity. Reuters Turk pledged to press AI firms to reduce AI risks.
The head of the Bank for International Settlements said in September 2026 that the AI boom poses new financial stability risks. Reuters The bank estimated the world's five largest technology firms will invest more than $1 trillion in AI between 2025 and 2026.
These warnings are about different dangers. Turk talked about harm to people and rights. The BIS talked about money risk, power held by a few firms, and huge spending on computers. Both came from outside the labs and pointed to missing rules, not to one single model.
The broader context here is that inside news now moves faster than official rules. A quit letter, a risk number posted online, and a podcast talk can shape views before a boss's plan or a government statement comes out. Engineers feel this, because launch choices and safety checks stay inside firms while public views form outside.
Looking at what this means for builders, the useful questions are smaller than the headlines. What sits behind the 10% guess. What safety checks, like crash tests for cars, would show it is wrong. What launch limits and checks after launch would lower risk. What public facts would let outside experts check the work. You do not need to believe the number to ask this. You need clear thinking that others can check.
In my view, the money number fits the same talk. More than $1 trillion from five firms in two years puts computers and skilled workers in few hands. That shapes which safety work gets paid for and which problems get fixed first. Money risk and loss-of-control risk are not the same, but both grow when things get very big without strong outside checks.
It is worth flagging what we do not yet know. We know about the quit, the risk numbers, the model name and timing, the podcast talk, the coming plan from Amodei, and the two outside warnings. We do not have test results, system cards, or details of Amodei's plan. Readers should see the 10% as one person's belief, not a measured fact.
The longer view here is that this could help build safer AI. Clearer tests before sale, better locks for AI that can improve itself, and buyers asking for proof of safety would help. My two kids grew up as seat belts and spam filters quietly became normal. Safety steps that lasted were built into everyday tools. AI safety will likely go that way, from loud warnings to quiet safeguards.
