California Wants an Off Switch for Very Powerful AI

California Gov. Gavin Newsom signed an executive order on September 18, 2026 calling for a "kill switch" for frontier AI models, the largest and most powerful AI systems. Like a circuit breaker that cuts power when wiring fails, the idea is a stop mechanism for an AI program that goes out of control. The order does not build it. It tells the state to speed up independent oversight and develop the tool, with outside reviewers checking over time if it works.

The order creates a group of national experts to write new AI safety recommendations within about two months, according to the most recent account of the plan Engadget. That group will send ideas to Newsom's office to update state AI safety laws. A separate account lists November 16 as the due date KCRA.

The order is about carrying out laws already passed. It aims to move faster on two AI laws Newsom recently signed, and builds on a new law that set up first-in-the-nation independent oversight of AI companies and safety checks Governor's Office. The administration said the September action directs work on the dangers shown in recent AI incidents. The "kill switch" is meant for AI programs that go rogue Los Angeles Times.

For builders, the main point is regular checking, not only shutdown. The order calls for a "kill switch" with its effectiveness checked on an ongoing basis by independent verification. It proposes letting "designated independent verification organizations" work inside AI labs for periodic audits and evaluations. That is different from point-in-time red teaming, a one-time attack test, or voluntary model cards, company-written safety notes. It would mean steady access, tests that can be rerun, and authority to check controls in lifelike conditions.

The panel will also look at reporting rules. That includes making AI companies file reports and risk reviews with an outside group, and disclose "loss-of-control incidents." The published summaries did not define those incidents. Scope, cutoffs, reporting deadlines, and handling of private test data were left for the expert process.

For context, in 2024 Newsom vetoed a bill that would have required pre-release safety testing and a kill-switch emergency shutdown for certain large AI systems Politico. The current order returns to similar tools through executive direction and expert recommendations rather than an immediate legal mandate.

Looking at what this means for labs and platform teams, the hard part is that models do not live in one place. Stopping a model used through an online service is easy to describe. Pulling back retuned copies, smaller copycat versions, saved model files, on-device copies, and copies hosted by others is not. Frontier systems rarely exist as one file in one data center. Any workable plan will need to say what stops: serving answers, further training, independent tool use, making copies, and who can order it.

In my view, the testing plan may matter more than the switch language. Like building inspectors who visit over time rather than once, outside checkers would need saved copies of each version, records of what the AI did step by step, shared lists of incident types, and tests that can be repeated. That can be done. It is also costly to run, and raises familiar questions about business secrets, cheating on tests, and keeping auditors independent.

Worth flagging for the November recommendations, reporting will depend on clear definitions. Without a narrow definition tied to observable behavior such as gaining access it should not have, trying to send data out, copying itself outside its safe area, or refusing a human shutoff command, reporting could become too broad to follow or too vague to enforce. With a clear definition and protection for companies that report honestly, California could get something it now lacks: a shared record of how frontier systems fail.

The broader context here is step-by-step lawmaking. California is trying to pair legislated oversight with a faster expert track that can update safety rules as capabilities change. If the panel writes shutdown rules and audit steps that can be tested and that work with real systems, other states and business buyers will use them. That would be a practical gain, even before any switch is ever used.