Power & systems

Killer robots: human safeguards cut the day before the deal

And this is the text Geneva is deciding on this week.

6 min read
Killer robots: human safeguards cut the day before the deal

What the text lost the day before it was adopted

Killer robots, as campaigners call them, are weapons that, once activated, identify, select and strike a target without a human stepping in. Since 2024, a group of government experts convened under the UN has been trying to agree common rules for them. On September 4, it adopted its final report by consensus.

We compared the draft report dated September 3 with the version adopted the next day. The draft still contained specific sentences on the human role. Several are gone from the final text:

  • that legal obligations be assessed by a human being, and ethical considerations taken into account;
  • that target and mission parameters can't be substantially changed without a human's authorization;
  • that these weapons operate in a way that is predictable and traceable back to the people responsible;
  • the ban on using them if their effects can't be anticipated and limited.

Another sentence changed its verb. States "must" set up internal mechanisms to handle incidents; in the adopted version, they are "encouraged" to.

What remains is the principle: human control and judgment are necessary, exercised "directly or indirectly". The handrail is still there, but several rungs have been taken out.

Who got these cuts? The group's final negotiations weren't public. Human Rights Watch, which followed the talks, writes that some states, including Russia and the United States, weakened the text by demanding changes to provisions that had broad support.

A blank in square brackets

The conference that opened Monday at the Palais des Nations in Geneva builds on that report. The Convention on Certain Conventional Weapons is a UN treaty that already regulates, protocol by protocol, weapons such as mines and incendiary weapons. Every five years, its member states meet for a review conference. The seventh runs until Friday, November 20, chaired by Swiss ambassador Julien Thöni.

The text meant to close the week is nearly done. Universalization, finances, the 2027 meeting calendar: most of it is written. Except for one subject.

In the draft final declaration the UN circulated on November 13, at the spot where countries are supposed to say what they'll do about autonomous weapons, you find only "[Placeholder on LAWS]". LAWS is the acronym for lethal autonomous weapons systems. The 128 countries have until Friday to fill that gap.

An instrument, but which kind?

The September 4 report runs to about twenty paragraphs that define these weapons, recall that the law of war applies to them in full, and list measures for human control. The UN presents them as "elements of an instrument, without prejudging its nature".

That phrase is the key to the deal. An "instrument" can be a treaty that binds its signatories, or a statement of good practice that binds no one. By declining to choose, the group made it possible for everyone to sign. This week's conference has to make that choice.

The debate over military AI is often framed as a company against a government, as in Anthropic's standoff with the Pentagon. Here, it's states that have to say which rules they'll accept for themselves.

Two texts, and a rule that weighs heavily

Two draft mandates are posted on the conference page. The first, with no author named, asks the group to "negotiate a new protocol" to the Convention, which means a legally binding text, and to deliver a draft in 2028.

The second comes from the United States, in a working paper dated November 4. Washington praises the work done, then proposes developing "a non-binding instrument". A version 1.0 would be adopted in 2029, with a final report due in 2031.

Its arguments: a non-binding text applies to everyone immediately, can be updated at the pace of the technology, and doesn't wait years for ratifications. The paper adds that it has "not been established" that these weapons form a category warranting new prohibitions. In September at the Security Council, Washington had already turned down the AI chiefs who came asking for rules.

The gap between the two is like the gap between traffic law and a road-safety leaflet. Both say stop at a red light. Only one says what happens if you don't.

The decision rule favors caution. The experts' report recommends that the conference decide on next steps "by consensus", and both proposals also envisage future work by consensus. With consensus, nobody counts votes: one country's objection is enough to block everything.

France wants to negotiate, without saying what

Le Monde headlines on Monday that France is "trying to open the door a crack" to regulation. The French foreign ministry's statement of September 10 shows which door that is. (The quotes below are our translation from the French-language text.) Paris wants the conference to adopt "a new negotiating mandate" for the expert group. The stated aim: "to start discussions toward a future international instrument that would ban fully autonomous weapons systems and regulate systems with autonomous functions".

The statement uses none of the words "treaty", "protocol" or "legally binding". It ends on the wish to find "a mandate that can rally consensus", meaning one acceptable to those who don't want a binding text. Automated Decision Research, an observatory linked to the Stop Killer Robots campaign, lists France among the "undeclared" countries on the question of a legally binding instrument. France has nonetheless signed the joint statements judging the text ready for negotiation.

Negotiate, yes. Say whether the result will bind, not yet. That reserve lets Paris talk to the backers of a protocol and to Washington alike, and it may be the best description of what a door left ajar means.

A majority isn't a consensus

According to the same observatory's count, updated September 4, 76 states have said they support opening negotiations on the basis of these elements. Seventy of them are parties to the Convention, a majority of its 128. In August, the UN Secretary-General and the president of the International Committee of the Red Cross renewed their call for a legally binding instrument and described the conference as a crucial opportunity.

But a majority can't fill an empty box. Without a common formula by Friday, the conference could break up with no negotiating mandate, or with one that extends the discussion without settling what kind of text it is.

By Friday we'll know whether the blank in brackets gets filled with a verb, "negotiate", or an adjective, "non-binding".

Was this article useful?

Topics covered:

RegulationGeopoliticsAnalysis

Frequently asked questions

What are the killer robots being discussed in Geneva?
They are weapons that, once activated, identify, select and strike a target without a human stepping in. The UN calls them lethal autonomous weapons systems (LAWS). The seventh review conference of the Convention on Certain Conventional Weapons opened on November 16 at the Palais des Nations in Geneva and runs until November 20. In the draft final declaration circulated on November 13, the space for autonomous weapons was still an empty placeholder in square brackets.
What disappeared from the experts' text in September?
Comparing the draft report dated September 3 with the version adopted by consensus the next day shows that several passages are gone: the assessment of legal obligations by a human being, the requirement that a human authorize changes to target and mission parameters, predictability and traceability back to the people responsible, and the ban on using these weapons when their effects can't be anticipated and limited. States that "must" set up mechanisms to handle incidents became states "encouraged" to do so. What remains is the principle of human control exercised "directly or indirectly". The negotiations weren't public, so it isn't known who got these cuts: Human Rights Watch writes that some states, including Russia and the United States, weakened the text.
What proposals are on the table in Geneva?
Two draft mandates have been published. The first, with no author named, calls for "negotiating a new protocol" to the Convention, in other words a legally binding text, with a draft in 2028. The second, a US working paper dated November 4, proposes a non-binding instrument, with a version 1.0 adopted in 2029. The experts' report recommends that the conference decide on next steps by consensus, which means a single objecting country can stop anything from passing.
Does France back a treaty on autonomous weapons?
France's foreign ministry statement of September 10 (our reading of the French-language text) wants the conference to adopt "a new negotiating mandate" toward a future international instrument that would ban fully autonomous weapons systems and regulate those with autonomous functions. But it uses none of the words "treaty", "protocol" or "legally binding", and it aims for a mandate that can rally consensus. Automated Decision Research, an observatory linked to the Stop Killer Robots campaign, lists France among the "undeclared" countries on a legally binding instrument.
How many countries want to negotiate rules?
According to Automated Decision Research's count, last updated September 4, 76 states have said they support opening negotiations on the basis of the report's elements, including 70 parties to the Convention, a majority of its 128. But a consensus decision isn't won by majority: without a common formula by November 20, the conference could end with no negotiating mandate, or with one that doesn't settle what kind of text it is.
Katja Liersch

Katja Liersch

Co-founder & Journalist

Katja is a journalist and TV producer. With decades of experience in mainstream media, she brings to Declic Media the perspective of those discovering AI: curious, demanding and pragmatic. She ensures every piece of content truly speaks to everyone.

All articles by Katja →
The free AI newsletter