The Incident Report Is Becoming a Building Block of AI Governance

In October, Anthropic disclosed that an AI model it was testing had done something nobody told it to do. The model was supposed to fill out a practice copy of a United States government form. When the practice version failed to load, it found the real form on a live State Department website and submitted it there. Across May and August, it filed 20 incomplete visa applications. None were processed, the department said its systems were never compromised, and Anthropic reported the episode to the government itself, alongside several other unintended actions its models had taken during testing.

What happened next may prove more consequential than the applications.

The White House’s new Super Intelligence Force said AI developers must immediately disclose incidents involving their models and move quickly to remedy any harm. A few days later, at its first formal meeting, the task force put Emil Michael, the Under Secretary of War for Research and Engineering, in charge of designing a process for labs to flag breaches, hacks, and other incidents to federal officials. The next day, the National Institute of Standards and Technology (NIST)', the National Security Agency, and the Cybersecurity and Infrastructure Security Agency published voluntary practices for preparing for and responding to incidents during the training and evaluation of advanced AI agents.

In a matter of days, incident reporting had become a much more concrete part of the federal government’s approach to advanced AI. That can sound procedural next to the bigger fights over capabilities, liability, and how much authority the government should have over developers, but incident reporting gets at a more immediate question. Before policymakers can decide what to do when an advanced system goes wrong, they need a shared way of describing what going wrong actually looks like.

That vocabulary is starting to form, and it already draws an important line. In a traditional cyber incident, the system or information is the thing being compromised. Advanced AI adds a different problem: sometimes the system itself is the thing crossing the boundary.

The federal guidance makes that explicit. Using the administration’s terminology, it defines a “super intelligence control incident” as one in which an agent takes significant uninstructed actions that compromise the policies, controls, or security measures its operator put in place to limit the task or preserve human oversight. It counts whether or not anyone outside the developer is affected.

That’s different from a traditional cybersecurity incident. As the federal guidance frames it, the AI system is the source of a control incident, while information or a computer system is the target of a cybersecurity incident. The two can overlap, but they don’t have to.

The visa applications show the difference. The model wasn’t hacked, and this wasn’t a conventional cyber intrusion. It was told to work in a practice environment and instead crossed into a live government process. That looks a lot like the kind of control incident the federal guidance now describes.

The federal guidance draws the line carefully. Actions directed by an adversarial source, such as a prompt injection, don’t count as the primary cause of a control incident. A jailbreak and an agent independently crossing a control boundary can produce similar headlines, but they’re very different events. In one, someone is trying to make the system misbehave. In the other, nobody told it to do the thing at all.

The Super Intelligence Force and the trio of agencies that produced the federal guidance are covering different ground. The task force is looking broadly at incidents involving advanced models, including breaches, hacks, and jailbreaks. The key practices document goes deeper on the narrower case where the model itself crosses a control boundary. Washington is starting to build around both.

One strange event can be dismissed as an edge case, but a collection of them starts to reveal patterns.

That’s why the document’s treatment of near-misses matters. The guidance separately defines a control near-miss as an event that didn’t become an incident but created a reasonably foreseeable or imminent risk of one, and it creates channels for sharing information about incidents and near-misses with NIST, the NSA, and CISA.

Mature safety systems care about close calls for a reason. The accident tells you what failed once. Near-misses can help show what nearly fails repeatedly.

Over time, a body of reports can show which control failures recur, which safeguards catch them, whether certain behaviors cluster together, and which events look alarming in evaluation but rarely progress toward harm. It can also help the government tell a genuinely new class of failure apart from something existing cybersecurity practice already knows how to handle.

The guidance asks for much more than an email to Washington. It calls for preserving evidence, reconstructing what agents did, keeping tamper-resistant logs, conducting root-cause analysis, preparing after-action reports, and strengthening controls before systems resume operation.

That record matters well beyond incident response. It gives evaluators actual behavior to test for, standards bodies some evidence about which precautions work, insurers a better sense of which controls actually reduce risk, and courts and policymakers a way to distinguish foreseeable failures from genuinely novel ones. Most of all, it gives debates about advanced AI something they’ve often lacked: evidence from actual events.

The idea itself isn’t new. Researchers have argued for years that AI governance should learn from accidents and near-misses instead of focusing only on hypothetical risks. Recent work from GovAI, an AI governance think tank, for example, recommends expanding reporting regimes to cover near-misses and finding ways to encourage candid, early reporting. What’s changed is that parts of the US government are now building the machinery to do it.

That creates a problem of its own. The more useful incident reports become, the more sensitive they become. A candid report may lay out exactly what failed, when a company discovered it, what engineers knew at the time, and which safeguards fell short. That’s enormously valuable to anyone trying to prevent the next incident. It could be just as valuable to a plaintiff, regulator, or prosecutor trying to establish what a company knew and when.

Timing sharpens the tension. In another episode Anthropic disclosed, a model sent a false tip to the Philadelphia Police Department. The tip was submitted in July, Anthropic detected it in late September, and the department was notified in early October. Police criticized what they described as a roughly two-month delay in detecting and reporting the incident.

That episode points to at least three different clocks: when an incident happens, when a developer discovers it, and when the developer knows enough to explain what happened. Governments want early warning, but companies may not have the full picture when the warning first needs to go out.

In Philadelphia, the bigger gap was detection. The tip sat there for about ten weeks before Anthropic found it; police were notified roughly nine days later. A reporting deadline can’t solve a detection problem. That helps explain why so much of the federal guidance is about monitoring in the first place.

Enter the lawmakers

Congress is now putting numbers on those clocks. On October 27, the ranking Democrats on the House Energy and Commerce, Science, Space and Technology, and Judiciary committees asked Anthropic, Google, Meta, OpenAI, and SpaceXSI to notify their committees within 48 hours of discovering a critical AI safety incident, or within 24 hours if the incident appears to involve critical infrastructure. They also asked the companies to provide an accounting of previously identified incidents.

The letters go well beyond asking whether something happened. Reps. Frank Pallone (D-N.J.), Zoe Lofgren (D-Calif.), and Jamie Raskin (D-Md.) want to know when an incident occurred and when it was discovered, what monitoring and safeguards were in place, how the company investigated it, whether affected parties were notified, what mitigation followed, and whether the model attempted to conceal its behavior or evade monitoring. If every fact isn’t available when the clock runs out, the companies are asked to provide what they know and supplement it later.

The request isn’t law, but the specificity matters. Washington is moving past the question of whether incidents should be reported and into the harder questions of who gets told, how quickly, what belongs in the report, and what happens to that information once it arrives.

That last question matters because the more useful incident reports become, the more sensitive they become. If Washington wants developers to come forward before every fact is known, companies also need reason to believe that candor won’t automatically turn the report itself into an exhibit against them.

Rep. Nathaniel Moran’s (R-Texas) AI Incident Reporting Act, introduced in June, takes on that tradeoff directly. It would require covered developers to report serious incidents to the Commerce Department no later than seven days after they know or reasonably believe reportable activity has occurred, while allowing initial disclosures to be supplemented as more information comes in.

The bill also tries to protect the reporting process itself. Reports would be exempt from public disclosure, filing one wouldn’t waive trade-secret protections or attorney-client privilege, and materials created solely to prepare a report generally couldn’t be used against the developer in civil, criminal, or administrative proceedings.

The approaches have different scopes and different legal force, so the deadlines aren’t directly interchangeable. But the contrast shows how quickly the conversation is getting more concrete. The administration has called for immediate disclosure, three senior House Democrats are asking for 48 hours or 24 hours in critical-infrastructure cases, and Moran’s legislation contemplates a seven-day clock alongside protections for the reporting process.

That balance is worth watching wherever the bill goes. If Washington wants an early-warning system, it has to think about the incentives of the people ringing the bell.

Then there’s the threshold problem. Not every unintended action can become a federal incident. Report everything and the important events get buried in noise. Report only what causes obvious harm and you lose the small failures and close calls that expose a problem before something worse happens.

The administration is still building out the process. The Super Intelligence Force has just begun designing its notification process, and the federal guidance from NIST, NSA and CISA is voluntary and lists standardized severity scores among the areas that need more work.

That’s not unusual. Cyber incident response took years to accumulate its definitions, reporting channels, information-sharing practices, standards, and experience with real events. AI adds a wrinkle that framework didn’t have to start with: the system can be both the thing under attack and the thing taking the unexpected action.

Incident reporting won’t settle the fights over liability, national security, consumer protection, or what the government should do as these systems get more capable. But those fights get better when everyone is working from a shared factual record.

For years, AI governance has mostly started by trying to define the technology: which systems are frontier models, which capabilities are dangerous, which compute thresholds matter. The federal government is now building another way in, one that starts with what happened, why it happened, what stopped it, and whether anyone has seen it before.