Microsoft's CEO calls for an emergency brake on AI

Saturday: a brake and a fear

On Saturday, October 10, Satya Nadella published a long post on X about the safety of AI models. The Microsoft CEO says "an authorized person" should always be able "to pause or shut down a model mid-task." His line, picked up by TechCrunch: "We must assume a model is compromised and contain it from the start. Think of it like an emergency brake."

The same day, a few hours earlier, the Guardian ran an interview with George Williamson, director of the Alan Turing Institute, the UK's national AI institute. His worry fits in a handful of words: that his country ends up relying solely on systems "that could be restricted or even switched off."

The brake one man asks for looks a lot like the kill switch the other dreads. The two men aren't answering each other and don't cite each other: we're the ones putting their texts side by side. But they describe the same mechanism, seen from opposite sides of the handle.

What Nadella is asking for

Nadella starts from an engineer's observation. With traditional software, you could trace a behavior back to the code that produced it. With today's models, you can't, yet we hand them sensitive data and critical actions. Hence his call "to step back and assess the trust architecture."

In practice, he proposes "separating the model from the harness that orchestrates its work", meaning the software that feeds it tasks and carries out its decisions, and "externalizing controls and safeguards." Every "meaningful model action" should leave "tamper-proof human readable evidence." Think of an elevator: the cable can snap, but the safety brake doesn't depend on the cable.

Nadella writes "Super Intelligence" for these systems, and TechCrunch notes it's the Trump administration's preferred term. The same post also calls for incidents to be reported quickly to the people they affect. It comes a day after Anthropic published an incident whose reporting delay Philadelphia police judged unacceptable.

What Williamson fears

George Williamson doesn't come from the startup world. He previously ran HMGCC, a low-profile government body that builds gadgets for British intelligence agencies, not to be confused with GCHQ, the signals intelligence agency. According to the Guardian, Alan Turing himself is among its former staff. Williamson took the helm of the institute after the government, its main funder, demanded a strategic shift.

His tone is measured, and his qualifiers matter. "We shouldn't depend solely on systems we can't inspect or control, or that could be restricted or even switched off," he says, with defence, national security, critical infrastructure and healthcare in mind. He adds that "both political and commercial pressures" could cut the country off from AI at moments when it wants access.

Then comes the sharpest line: "There's a set of circumstances where it would be a good idea for us to have something that we don't have to ask other people's permission to use." Some circumstances, not all. Williamson isn't calling for self-sufficiency at any price, and he says the institute won't try to copy the frontier labs.

The brake has already been pulled

The emergency brake isn't hypothetical. It was pulled at scale on Friday, June 12. That day, to comply with a US administration directive aimed at foreign nationals, Anthropic took Fable 5 and Mythos 5, its most advanced models, offline, according to the Associated Press. We've already covered how it got there, then the conditions under which the model came back. Anthropic announced the controls had been lifted on June 30, eighteen days later.

What the episode says about the handle is simple. Neither the user nor the provider decided to pull the brake: Anthropic shut its models down on orders. It said so at the time, without disputing the principle: "We believe the government should have the ability to block unsafe deployments, as part of a statutory process that is transparent, fair, clear, and grounded in technical facts. This action does not adhere to those principles."

That's the precedent the Guardian points to in its Williamson interview. Seen from London, Nadella's "authorized person" already has a face, and works in a foreign ministry.

Who holds the handle

An emergency brake has two sides. For whoever holds the handle, it's a safety feature. For every other passenger, it's a dependency: the train stops when someone else decides it does. Nadella's post doesn't say who should hold the handle at national scale, because it's about something else: companies deploying agents in-house.

Still, two of his principles point in Williamson's direction. Nadella writes that no model should become the sole dependency for an important outcome, and that organizations should decide independently what a model can access and what it can do. Transposed to a country, that's almost the Alan Turing Institute's agenda.

None of this makes the brake undesirable. When Congress moved to impose a stop button on AI labs, we noted the real question was whether it actually cuts, and fast enough. Williamson adds a second one: cuts on whose behalf.

A company can choose, a country much less

A company can apply Nadella's principles fairly quickly: several models, controls placed outside the model, logs you can reread. It's a matter of architecture, and the company stays in charge of its choices.

A country has less room. Williamson acknowledges the UK probably can't match the financial resources of the big labs, and bets on smaller models and nationally owned intellectual property, made available to startups. He isn't alone in asking the question: the Guardian notes that Christine Lagarde, at the European Central Bank, has spoken of the threat of being "cut off" from US and Chinese technology.

Nadella's brake protects whoever pulls it. Williamson is thinking of the people sitting in the carriage: "It feels pretty uncomfortable if you don't have some control or ability to use technologies when you want them, and also harder to imagine how you build an economy on the top of stuff that can disappear."