Could a Judge Stop a Dangerous AI From Coming Out?
Anthropic has written a draft U.S. bill that would let a judge stop an AI model from being released in specific disaster cases.
An Anthropic spokesperson confirmed the draft and said it uses ideas from the company's Advanced AI Framework, which it has discussed with lawmakers since June, according to reporting published Oct. 20, 2026. TechCrunch
News of the outreach first appeared on Oct. 16, 2026, when Reuters, citing Politico, reported that Anthropic had quietly pitched its own AI safety bill to lawmakers. Reuters The details described on Oct. 20 call for testing before release plus a court order, not power for an agency to act alone.
Under the draft, outside experts approved by the U.S. Commerce Department would test a model for disaster risk. Like building inspectors who check a structure but cannot order work to stop on their own, they would report what they find. An expert who finds an imminent catastrophic risk would have 72 hours to tell the Department. AI labs could pay the experts, but pay could not depend on the result. If told, the Commerce Department would have to ask a judge in Washington, D.C., for an order to halt release. The draft says no federal agency may halt distribution by itself.
The limit is strict. A stop requires a foreseeable, imminent risk in which more than 50 people could die or more than $1 billion in property damage could result. The risk must also be one of three types: creation or release of a biological weapon, the model escaping its maker's controls, or a cyberattack. General misuse, bias, job loss, or smaller safety failures would not count under this power.
The numbers match California law. California Senate Bill 53 uses the same 50 deaths and $1 billion figures to define catastrophic risk. SB 53 also requires builders of the most powerful AI models to publish a plan for how they check and handle those risks. TechCrunch Anthropic has published such a compliance framework for SB 53. Anthropic
The draft follows other Anthropic work on risk. That includes an October 2024 post titled 'The case for targeted regulation' about cutting disaster risk while supporting new ideas, an August 2026 risk report on whether its systems pose catastrophic risk in several categories, and a September 2026 threat report on threat actors it disrupted between December 2025 and August 2026 across seven areas of harm, from cyber operations to biological misuse.
The broader context here is a single path to a pause: outside testing, referral by the Commerce Department, and review by a federal judge. That setup tries to catch danger early without giving every agency power to stop a release, with a court as a check. Payment by the developer keeps testers able to see the models, but puts weight on who can serve as a tester, how independence is kept, and what happens if a risk is missed. Open questions include which models are covered, what access testers receive, what proof meets the test of foreseeable and imminent, and how a D.C. court handles fast, highly technical requests kept private. The 72-hour rule and the death and damage counts would need clear ways to measure. Counting deaths that might happen, or tracing damage from a general tool tested before use, is hard.
In my view, a bill written by a company that would face the rules will get hard questions, even if it is narrow. That does not remove the interest of the idea. A clear court path limited to these three dangers with fixed numbers is easier to follow than broad agency power. If some version moves forward, it would give safety teams something concrete to build for, with records, tests, and pre-release pauses tied to outside law rather than company policy alone. That kind of shared standard could help steady safety work grow over time.
