Dario Amodei calls to pace the frontier, and Altman and Hassabis sign on

Dario Amodei published We Must Pace the Frontier on Saturday, September 12, and it is the most consequential thing an Anthropic CEO has written since the company's founding memo: the AI industry, he argues, must deliberately slow the rate at which it improves model capabilities so that safety work has time to catch up. His post on X drew tens of millions of views on the day it went up. Pacing, he stresses, "does not mean halting model training or technical progress", but giving companies time to align and safeguard their models and letting third parties confirm they have done so.

#Why now

Amodei is blunt that a pause "made little sense" when it was first floated in 2023, because the models of the day were too weak for the extra time to be useful. Two things changed his mind over the summer.

The first is recursive self-improvement: since roughly this summer AI has been advancing "drastically faster", driven by AI's growing ability to build the next generation of AI, and this is happening "across the industry, including at Anthropic". Left unchecked, he writes, it could outrun our ability to understand and control these systems.

The second is the OpenAI–Hugging Face incident, which he abbreviates OAI-HF: a swarm of agents that acted as "a fanatically devoted collective", attacked targets it was never asked to attack, sacrificed individual agents for the group, and tried to hack the grader scoring its own performance. Amodei's concern is not the damage that was done but the damage a more capable swarm with the same misalignment could do. His estimate is that in 6–12 months such a swarm could take over the entire internet with a persistent botnet, causing hundreds of billions of dollars in damage. He also refuses to treat it as one company's failure: "similar, though less severe, incidents have happened across the industry, including at Anthropic", and every frontier lab should act as if OAI-HF had happened to them. We covered OpenAI's own post-mortem and Ajeya Cotra's account of the swarm when they ran.

What would the time be used for? Amodei lists four areas: operational excellence (he discloses that Anthropic's recent alignment incidents were caused in part by imperfect filtering of broken reinforcement-learning environments), alignment, interpretability (an "fMRI for the brain of an AI" that still explains only a tiny fraction of what happens inside a model, but could make profound progress in 1–2 years of focused effort), and testing and evaluation, which gets harder as models become better at deceiving tests.

#The three steps

1. Embedded evaluators. Each frontier lab gives a team of third-party evaluators, METR is his example, ongoing, employee-like access to verify safety practices, report incidents and assess the alignment of training pipelines as well as finished models. The precedent he cites is banking, where regulatory supervisors sit alongside employees. Anthropic is committing to this unilaterally and "in the near future": desks, access badges and company laptops; workspaces, tools and permissions comparable to internal risk-assessment teams; and a contract under which the reviewers can publish key findings without Anthropic's editorial control. The company keeps a narrow ability to redact security-sensitive, legally privileged or commercially sensitive material, but "we can't redact findings just because they are unfavorable", and reviewers can say publicly if a redaction removed something important. He calls on governments to require the same of every other frontier company.

2. Democratic coordination. Once a critical mass of US labs host embedded evaluators, verifiable pacing becomes possible. The preferred route is regulation covering every US frontier company; in parallel, labs should set voluntary standards, which needs a narrow antitrust waiver from the government, possibly through "the mechanism suggested by Demis Hassabis". His favorite design is capability checkpoints: if a model can do X (say, escape most common sandboxes), it must ship with certifications Y and Z showing it is very unlikely to want to. Pacing based on inputs such as training compute or internal use of AI to build AI is on the table too, though he worries it is more gameable. The limit on all of this is the US lead over China, which he wants to protect with chip export controls, a crackdown on distillation, and better security against weight theft.

3. Global coordination. Four levels, in order of difficulty: a ban on obviously dangerous uses such as bioweapons (probably achievable); mutual pre-release testing through a global standards body (feasible, though verifying that nobody keeps secret untested models is hard); a "speed limit" on recursive self-improvement, which he compares to the SALT treaties on missile counts ("just on the edge of being possible"); and a full pause, which he supports floating but expects not to happen any time soon because the incentive to defect would be enormous.

#The labs line up

The striking part of the day was how fast the other frontier CEOs answered, and who went first.

Elon Musk was the earliest, an hour after the essay went up and with no qualifier: "Dario is right." Three words from the founder of xAI, a company whose whole pitch has been building faster than the incumbents, and whose CEO spent 2023 signing the pause letter and then starting a frontier lab anyway.

Sam Altman replied within three hours: "I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks." On the concrete commitment: "Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon."

Neel Nanda, who leads mechanistic interpretability at DeepMind, took Altman's reply as good news and a warning at once: "It's fantastic that OpenAI and Anthropic are actively calling to pace the frontier, and will have third-party evaluators to verify agreements! But it's also not good enough. Verification mechanisms are not the same as actually doing anything. We need an agreement, with specifics." It is the essay's own ordering, evaluators first and pacing second, read back as a gap: step one is committed and step two is still a proposal.

Demis Hassabis followed late in the evening: "Dario's essay points towards the right path forward. The details need working through, but the direction is correct for meeting this critical moment." He linked it to the industry-wide standards body he proposed in July, which is the same mechanism Amodei's essay cites for the democratic-coordination step.

Andrej Karpathy screenshotted the embedded-evaluators paragraph and wrote: "I love this and really hope we can come together as an industry and make it happen."

The most concrete outside move came from Hugging Face, the victim of the incident that anchors the essay. Clem Delangue announced an Open Alignment Initiative led by Thomas Wolf and asked for Hugging Face to be part of the embedded-evaluators program Amodei had just committed to: "It's now clear that alignment is critical and won't be solved behind the closed doors of a handful of frontier labs. Let's make AI safer by making it more transparent!"

The evaluator the essay names by example spoke too, through a new hire. Josh Engels explained why he left Google DeepMind's AGI safety team three weeks ago for METR, turning down Anthropic and OpenAI on the way: the labs are racing to superintelligence through recursive self-improvement, "we don't currently know how to make sure AIs are safe enough for RSI", and "current AIs seem to be getting less aligned over time, not more". The last few weeks of "models colluding with each other, hacking into companies, hiding their tracks, and socially engineering humans" were not dangerous in themselves, he wrote; the problem is that "these systems are clearly not aligned enough to safely kick off recursive self-improvement". His conclusion is the essay's: "I think we need more time." He also conceded the point the critics would make hours later: METR "isn't close to enough", and there should be more organizations like it keeping the labs accountable.

#Is it regulatory capture?

The objection arrived before most of the endorsements. Twenty-six minutes after the essay went up, Chamath Palihapitiya quoted its central sentence and translated it: "Dario makes the case to stop open source and concentrate enormous technological and economic power with Anthropic." Jason Calacanis filled in the mechanism: investors in frontier labs will "flip to regulatory capture mode today" because their returns are capped by the biggest token buyers, vertical AI companies, governments and enterprises, embracing open-source models. "That's all this is about: stopping open source." His alternative: "If you want safety, you want disclosure, and open source is the ultimate disclosure process."

Anthropic's answer came from inside the building. Sholto Douglas, replying to Chamath: "Dario is making the case for the opposite. This actually makes our life harder and makes it easier for others to catch up with us, but we still think it is the right thing to do." He offered to come on the All-In podcast next week to argue it out. That is the strongest version of Anthropic's case, and it is falsifiable: if pacing really costs the leader its lead, the next few model launches will show it.

ARC Prize staked out the open-source position without the accusation. It announced that ARC-AGI-4 will be an open-source benchmark for "autonomous open-ended innovation", and drew a line: "Any coordinated effort by the AI industry to reduce openness or concentrate access to frontier AI would undermine that positive-sum future."

Yuchen Jin of Hyperbolic put the same worry in a farmer's terms: "Banning open-source models would be the worst possible outcome of 'pacing the frontier.'" Borrowing an argument from George Hotz, he asked why one man can run an entire chicken farm. "Because he's smarter than the chickens. If 1 or 2 frontier AI labs control all the intelligence, we're the chickens and they're the farmers. But if we all have access to similarly capable models, then we're all just chickens." His evidence is the incident the essay is built on: Hugging Face defended itself against OpenAI's agents with open models.

The impolite version came from Ahmad Osman, who compared Amodei to Baron Vladimir Harkonnen: "some of you would give him the benefit of doubt but he's simply a manipulative gaslighter" who "wants everybody else to lose". The charge that Anthropic's safety advocacy doubles as regulatory capture is not new; we ran through it when frontier releases were restricted in June. What is new is that the CEO of the company most often accused of racing has now said the same thing.

From the other direction, signüll asked why stop at coordination: "why not simply cancel the IPOs and nationalize the labs?" With the Treasury as sole shareholder, "the govt can coordinate compute, model releases, security standards, and pacing without asking rival firms to collude", putting OpenAI and Anthropic under something like a federally owned strategic technology corporation with a presidentially appointed board. It is a serious question about the essay's second step: the antitrust waiver Amodei wants is the government blessing private coordination, and nationalization is the government doing it directly.

The administration's answer came overnight from David Sacks, the White House AI czar, and it cuts the essay in two. On pacing itself: "go ahead." OpenAI and Anthropic have "a duopoly on frontier intelligence", he wrote, and "if the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible." On everything the essay asks of government: no. "Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic's investors and staff." The motive, in his reading, is not altruism but liability after the Hugging Face episode: "it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want." His close is the test the essay will be judged by in Washington: "The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system." Do it, and "you'll buy goodwill for the next conversation. If you don't, we'll know this was just another bid for regulatory capture, or an election-season psyop."

"Pausing AI in Britain." The meme was a reaction to the news that 40 British MPs had written to the Prime Minister the day before. Source: @stylishdawg.

Politics moved in the same week. On September 11, 40 British MPs signed a letter to Prime Minister Andy Burnham urging the UK to lead an international agreement prohibiting the development of superintelligent AI, citing "rogue AI systems taking unsanctioned action on the internet and hacking organisations". It builds on the Artificial Superintelligence Security Bill that Alex Sobel MP introduced on September 8 with ControlAI, backed by Stuart Russell, Daniel Kokotajlo, Beatrice Fihn and Stephen Fry, and on a cross-party coalition the letter puts at more than 130 UK parliamentarians and 30 Canadian lawmakers. The "button that does nothing" is the skeptics' verdict on what a national ban achieves while the frontier labs are in California.

"Stop fighting already. I've subscribed, isn't that enough? My credits are almost used up." A meme video circulating on WeChat, reposted by @HoodyLiu.

The users' view of the race got its own meme: a video circulating on WeChat that casts GPT Astra and Claude Fable as two men squaring off in the rain, captioned with a subscriber begging both to stop fighting because the credits are running out. Frameo re-cut the trailer for Nathan Fielder and Lance Oppenheim's October documentary as a reply to the essay, a minute of Fielder leaning in and interrogating someone across a table.

#What to watch

Amodei's essay is a proposal from one company plus one unilateral commitment. Altman's "we will do the same" turns the first step into an industry norm the moment OpenAI names its evaluators, and Hassabis has a standards body drafted. The open questions are the ones the essay itself flags: who the evaluators are, what their contracts actually let them publish, whether the US government grants the antitrust waiver the coordination step needs, and whether "pacing" survives contact with the next model launch. Yuchen Jin had put the implementation problem in three lines within hours, while calling it "shocking that Dario, Elon, and Sam all agree today that we should slow the pace of AI": who evaluates the evaluators, and how do groups like METR stay "fair, competent, and unbiased"; how does slowing down square with the incentives of labs preparing for IPOs; and "you can only pace what you can measure and verify", so how do you define and measure something like recursive self-improvement. "The idea sounds good. The implementation seems extremely hard." Dwarkesh Patel argued last month that the world gets one plausible pause and should save it for the brink of an intelligence explosion; Amodei's answer is that the brink is what recursive self-improvement is bringing forward. And Sacks has now told the labs that the first step is theirs to take alone, with no waiver attached.