AI Nearly Sparked a War: US Military Almost Seized a Chinese Ship Over a Hallucination

As CNN reports, citing four sources, this spring — at the height of the war with Iran — an intelligence report circulated through US military channels with an alarming conclusion: a Chinese vessel in the Middle East was allegedly carrying components for a nuclear weapons program.

The military reacted instantly. According to CNN's sources, an interception of the ship was being prepared: armed service members were set to board the vessel, and aircraft were already in the air. It was only just before the operation began that officers took a closer look at the document and discovered it had been prepared by an analyst at special operations command with the help of artificial intelligence — and that the chatbot had misidentified the nature of the cargo. What the vessel was actually carrying could not be established.

One source called the report "completely false," adding that it "almost started a war": any operation against a Chinese ship could have escalated into an armed conflict between the two powers.

How the False Conclusion Was Born

The analyst turned to the chatbot with a query about intelligence on a ship's manifest that had come in from US Special Operations Command Pacific in Hawaii. The bot combined open sources with classified signals intelligence data from secure repositories and reached its fateful conclusion about the nature of the cargo. The analyst then resorted to AI again — this time to package the result into the standard intelligence-report format the military is accustomed to trusting — and sent it out. It remained unclear whether a commercial chatbot or a government product was used; according to a former senior official, "the internal tools are basically the same commercial ones, just with lipstick on."

A Race Without Rules

The Pentagon is rapidly rolling out AI into literally everything — from analyzing raw intelligence and selecting targets to logistics and budgeting. In January, Defense Secretary Pete Hegseth unveiled an "AI Acceleration Strategy" meant to remove bureaucratic barriers and put "cutting-edge American models" into the hands of three million military and civilian personnel at every level of clearance.

The trouble is that the process is decentralized: different units use different tools under different rules, there is no single standard for verifying generated information, and the reliability of the systems varies widely. Sources note that AI is increasingly being used for targeting — an area where a mistake means people die — yet clear guidance on how a "human in the loop" would prevent civilian casualties or friendly fire simply does not exist.

Such "hallucinations," according to CNN's sources, are not isolated cases. Senior officers worry that AI is pushing analysts to produce material faster, and that younger staff who grew up on these tools trust them too uncritically. One source's bottom line sounds almost like an aphorism: "AI lets you get to a bad idea faster."

Source: Hacker News