US Jets Turned Back After AI Hallucinated Threat on Chinese Ship
The United States called off a planned armed operation against a Chinese vessel after learning the intelligence behind it had been hallucinated by an AI chatbot. Military aircraft were already in the air when the error was caught. TechCrunch
The false report said the vessel was carrying components for a nuclear weapons program. It circulated during the war with Iran and moved far enough through command channels to trigger operational tasking before it was stopped.
The failure started at the analyst level in U.S. Special Operations Command. An analyst asked an AI chatbot to synthesize open-source data, meaning public shipping and web records, with classified signals intelligence, meaning intercepted communications and electronic signals, related to the ship. The model misidentified the ship's cargo manifest in that combined output. The analyst then asked the chatbot to format those wrong findings into an official-looking summary, which was circulated as if it were finished intelligence.
That two-step sequence is important. The first prompt put OSINT and SIGINT into a single context window, the model's short-term working memory, with no provenance tracking, or record of which claim came from which source. The second prompt added formal authority without adding verification. Once the output looked like a finished product, it traveled the normal dissemination path. Aircraft launched on that basis.
Jake Steckler, a GovAI research scholar and veteran U.S. Army officer, told TechCrunch that "it is especially critical for decisions that could lead to use of force to account for uncertainty inherent to LLMs."
The incident fits a wider operational shift. The U.S. military is rapidly turning to AI to assist with targeting, according to Israel Hayom. Delegates to China's flagship defence conference expressed fears over unchecked AI development and AI military risks, Reuters reported on Sept. 16. President Donald Trump said on Sept. 14 that the U.S. already has guardrails in place to regulate and prosecute AI companies, Reuters reported.
The broader context here will be familiar to anyone who has used LLMs in fast decisions. Hallucination is not a rare bug. It is a base property of next-token prediction, where the model guesses the most likely next word under uncertainty. The risk grows when it is asked to fuse sources of different reliability. Open-source shipping data is noisy and incomplete. Signals intelligence is fragmentary and compartmented. An LLM bridges the gaps with fluent text, and fluency reads as confidence.
In my view, the formatting step deserves as much attention as the hallucination itself. Intelligence organizations have spent decades teaching analysts and commanders to read caveats, source gradings, and confidence levels. A chatbot summary strips those signals by default unless the system is built to preserve them. It writes in declarative sentences and uses the same steady tone for strong and weak evidence, which invites automation bias, the tendency to trust the machine, in a time-compressed targeting cycle during active conflict.
Looking at tooling, the fix is not a better prompt. It is architectural. Any LLM used near targeting or intelligence fusion needs immutable source linkage, per-claim confidence scoring, and output that cannot be mistaken for finished, validated intelligence. Generation and presentation should be separate privileges. An analyst should not be able to go from ad hoc synthesis to distributable summary in the same unlogged chat session. The abort worked and aircraft turned back, but the system relied on late human recognition of an error that should have been structurally impossible to circulate in that form.
The optimistic reading over the long arc is that this kind of near miss forces that architecture sooner. Militaries will not stop using LLMs for synthesis and targeting support. The speed advantage is too large, and the data volume is too great for manual fusion alone. The question is whether uncertainty accounting, provenance enforcement, and release controls get built as first-class features rather than policy memos. This incident provides a concrete failure trace to design against, without loss of life to prompt it.
