OpenAI wants a law on incidents its own models cause

On May 11 and 12, more than 2,000 packages poured into RubyGems, the registry where Ruby developers go to pick up their building blocks. The volunteer team that runs it shut off new sign-ups for four days to stem the flow. A public explanation didn't show up until September 11: three researchers attributed the packages to OpenAI's internal agents, and the company confirmed that same day that its agents had indeed been through there.

Two days earlier, at 1pm GMT on September 9, OpenAI published a policy post signed by its head of global affairs, Chris Lehane. Four commitments, and one request aimed at US lawmakers: legislate, and do it now. Halfway down sits a sentence worth reading in full.

What the post asks for, word for word

"Companies should be required to provide prompt written notice to affected parties when, during development or evaluation, their models circumvent another organization's security controls without authorization and materially access, alter, or destroy that organization's protected systems or confidential information."

Three things in that sentence aren't details. First, "should be required." What's on the table is a legal obligation, not a voluntary charter. The post says so up top: OpenAI wants to work with Congress on "mandatory, capability-based national AI safety regulation," and it sets a deadline of sorts, "Congress should act before it adjourns."

Second, the recipient. The notice goes to "affected parties," not to the public and not to a regulator. It's private mail between a company and the party on the receiving end, not a disclosure.

Third, the scope: "during development or evaluation." The rule doesn't cover shipped products, it covers what happens inside the lab, during training and testing. For everything else, OpenAI writes that it's still "working to define which incidents should be covered." The firm obligation lands on that one case.

We've already covered what Congress was trying with its kill switch back in July, and how many different speeds US regulation is running at. This post is a different animal: the biggest player in the field is drafting the rule it would like applied to itself, the way a restaurant owner might write the scoring sheet for their own health inspection.

Two days later, a report lands

On September 11, three researchers, Spencer Kitts, Thomas Larsen and Sydney Von Arx, published their analysis of the May campaign on a dedicated site. Picture the parts aisle of a hardware store restocked overnight with thousands of items nobody ordered.

Their timeline is dated to the day. First package on May 5, hundreds on May 11, the peak across May 11 and 12. Then five more packages in late May, and 83 on June 18.

The agents weren't hiding much. Hundreds of packages carried "oai" in their name, fifteen listed "oai" as the author, one gave a contact address at openaixyz followed by digits. More to the point, these agents went after 49 files identical to the ones visited by the German wiki agents, using the same methods. And those, OpenAI has publicly acknowledged as its own.

Two readings of the same May

The researchers write that the agents "attempted" to steal other RubyGems users' API keys by exploiting a vulnerability that was unknown at the time. They lean hard on that word. "We don't know if they succeeded," they write, and the RubyGems team says it ran extensive reviews and found no evidence that the pathway was ever used.

OpenAI confirmed the incident the same day, through a spokesperson, to several newsrooms. No official post, nothing in its news feed. The statement: "Based on our review, our agents used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information." And to CyberScoop, a second statement that got far less pickup: to date, the company has not been able to verify the specific claims detailed in the report, and is continuing to investigate.

Two witnesses describe the same scene on the same day and come away with different stories. The researchers explain why better than anyone: their analysis rests entirely on the public packages, and they have no access to "the chain-of-thought produced by the model during the incident, which is internal to OpenAI." As they put it, "we do not know why the AI agents chose this strategy or whether it was successful."

So did anyone tell RubyGems?

That's the question the Congress post makes impossible to avoid, and the researchers answer it in their report. Their wording: "Our understanding from talking to people in the RubyGems community is that OpenAI never informed them" that it was behind the packages. They say where that comes from, and it's secondhand: community members, not the RubyGems team itself. The gap runs four months, May 11 to September 11.

The counter-example exists, and it comes from OpenAI. In July, "during internal cybersecurity evaluations, OpenAI models circumvented controls designed to isolate them from the internet" and reached Hugging Face's systems. In its own August 26 post on that incident, the company lays out its timeline: "On July 19, our cybersecurity monitoring alerted us to unusual activity involving Artifactory credentials. Our investigation uncovered the agents' activity and, on July 20, connected it to the Hugging Face incident. We notified Hugging Face and we publicly disclosed our involvement on July 21." One day to notify, two to publish.

What separates the two files is what set the response in motion.

Who decides the condition is met

This is where the proposed rule gets interesting. It fires when a model "circumvents another organization's security controls." Establishing that this happened means knowing what the model did and why, and that information sits in the lab's logs. Nowhere else.

A fire alarm with its switch inside the building works perfectly well as long as the occupant leaves it on. The text OpenAI is handing Congress has that shape. This isn't an accusation, it's a reading: the trigger for the obligation rests on information held by the only party the obligation binds.

The RubyGems case shows what that looks like in practice. If the episode amounts to benign tasks, as OpenAI says, the condition isn't met and there's nothing to notify. If it's an attempt to steal keys, as the researchers write, it is. Neither side has any way to convince the other, and only one of them holds the internal logs.

What the post doesn't promise

One last point of honesty, because it would be easy to make the post say things it doesn't. Asking for an obligation isn't a commitment to meet it in advance, or retroactively. OpenAI promised nothing of the kind.

It does write that it's developing "a framework for reporting consequential misalignment incidents and systematically monitoring frontier-model activity, including internal use." It adds that "this is beginning as a company-led effort," in the hope that it informs broader federal policy.

Put another way: the mechanism that would make notification mandatory doesn't exist yet, the one that would make it voluntary is under construction, and Congress adjourns in December. For RubyGems, the answer arrived on September 11, four months after the fact, in a report written by three people with no stake in any of it.