Anthropic Wants a Judge to Decide When an AI Model Is Too Risky
Anthropic is circulating a draft federal bill that would create a court process to block distribution of an AI model in defined catastrophic-risk scenarios.
An Anthropic spokesperson confirmed the draft and said it contains proposals from the company's Advanced AI Framework that it has discussed with lawmakers since June, according to reporting published Oct. 20, 2026. TechCrunch
The outreach first became public on Oct. 16, 2026, when Reuters, citing Politico, reported that Anthropic had quietly pitched its own AI safety bill to lawmakers. Reuters The details described on Oct. 20 outline testing before release and a court order, not authority for an agency to order a takedown on its own.
Under the draft, independent evaluators registered with the U.S. Commerce Department would test a model's disaster potential. An evaluator here means an outside technical team approved to run safety checks, similar to outside auditors reviewing company books. An evaluator that finds an imminent catastrophic risk would have 72 hours to notify the Department. Evaluators could be paid by AI labs, provided payment is not tied to the result. If notified, the Commerce Department would be required to ask a judge in Washington, D.C., for an order halting release. The draft states that no federal agency may halt distribution on its own.
A block would require a foreseeable, imminent risk in which more than 50 people could die or more than $1 billion in property damage could result. The risk must also fall into one of three causes: creation or release of a biological weapon, the model evading its developer's controls, often called loss of control, or a cyberattack. General misuse, bias, economic displacement from automation, or less severe safety failures would fall outside this blocking authority.
The numbers match California law. California Senate Bill 53 uses the same 50 deaths and $1 billion in damage figures to define catastrophic risk. SB 53 also requires frontier developers, meaning builders of the most capable models, to publish a framework explaining how they assess and manage those risks. TechCrunch Anthropic has published a compliance framework for SB 53 covering that obligation. Anthropic
The draft follows other Anthropic publications on measuring risk. The company published principles for reducing catastrophic risks while supporting innovation in an October 2024 post titled 'The case for targeted regulation.' It published an August 2026 risk report assessing the extent to which its systems pose catastrophic risk in several categories, and a September 2026 threat-intelligence report on threat actors it disrupted between December 2025 and August 2026 across seven areas of harm, from cyber operations to biological misuse.
The broader context here is a single federal pipeline for a pause: registered outside testing, referral by the Commerce Department, and review by a federal judge. That design tries to provide visibility before release without giving each regulator its own power to stop distribution, while placing a check by a court on a consequential executive request. The tradeoffs will be familiar to readers who have worked with government security reviews such as FedRAMP or with time-limited vulnerability disclosure. Payment by the developer with a ban on results-based pay keeps testers able to access models and testing tools, but places weight on who qualifies as an evaluator, how independence is enforced, and what happens when a risk is missed. Practical questions include which models are covered, what access evaluators receive, what evidence meets the test of foreseeable and imminent, and how a D.C. court handles fast, highly technical requests filed under seal. The 72-hour clock and the death and damage thresholds would need clear measurement methods. Estimating deaths that have not yet happened or tracing property damage to model-enabled actions is difficult, especially for general-purpose systems tested before release.
In my view, a statute drafted by a vendor will invite close scrutiny, even when narrowly written. That does not remove the interest of the mechanism described here. A defined path to pause distribution through a court, limited to bioweapon, loss-of-control, and cyberattack cases with stated harm numbers, provides clearer limits than broad agency discretion. If some version moves forward, it would give safety and policy teams a concrete point to build toward, with logs, testing setups, and pre-release holds tied to an outside legal standard rather than company policy alone. That kind of shared interface is what makes consistent safety work possible over time.
