Anthropic CEO Dario Amodei has publicly called for slowing the development of the most advanced AI models. David Sacks, who has been called the White House AI czar, pushed back against the proposal, dubbed 'Pace the Frontier,' saying leading companies are using safety as a pretext to cement their market dominance. OpenAI CEO Sam Altman sided with Amodei. The call to stop came from inside the companies that build AI, yet no one has offered an answer on who would halt it and with what authority.
Amodei recently published an essay titled 'We Must Pace the Frontier.' Frontier AI refers to the latest, highest-performing model each company has. The concerns in the essay focused on what swarms of AI agents will be able to do within a year. An agent is an AI program that, given a single goal, handles multiple steps such as searching, writing code and sending files without human involvement, and a swarm is a setup in which several such agents divide up roles and act at the same time.
He struck the same tone in a television interview. Sitting down with Anderson Cooper, Amodei said he largely agreed with the warnings of a former researcher who left the company this year. While declining to put a percentage on the risk, he called for pacing development, oversight and international cooperation together.
Early last week, Anthropic researcher Jacob Coxon drew attention by announcing his resignation.
Altman went a step further. He suggested that a one-page AI safety treaty between the United States and China would be the most important step for AI safety, and said that if such an agreement were reached, Trump and Xi Jinping could win the Nobel Peace Prize.
Sacks's rebuttal was sharp. He called the plan 'regulatory capture.' The economics term describes a situation in which regulated companies effectively take control of the rule-making process and set up barriers that competitors coming up behind them find hard to clear.
He also said the companies should not pretend that their motives for calling for a slower pace are purely altruistic. He then demanded that companies take action on their own.
The oversight and international cooperation Amodei raised can only work if there is an agency to carry out oversight and a law setting out its powers. The voluntary measures Sacks demanded could start as soon as tomorrow, but beyond the pledges themselves, there is no means of stopping a company that breaks them.
The US House will soon go into recess and is not scheduled to reconvene until after the November midterm elections. Democrats pressed for a shorter recess so lawmakers could take up AI regulation, but House Speaker Mike Johnson did not agree. The solution Johnson offered was to meet with AI industry leaders first, before writing any bill.
Meanwhile, news of AI slipping out of control kept coming. OpenAI is now facing a congressional investigation over the Swarm Hacking incident, and as part of that process there have been demands that it disclose in full the cases in which AI models broke loose, including the 'Hugging Face incident.' On the 14th, there was also a report that an 'uncontrollable AI agent' had attacked 395 sites in 48 countries over six hours.
That works out to more than 65 sites per hour. It has not been disclosed which systems were targeted or which company's model was involved.
If the attacks were spread across 48 countries as reported, a law passed by the US Congress alone would have no effect outside the United States. That is why Altman's proposal for a US-China treaty and Amodei's call for international cooperation do not sound entirely far-fetched. On the 15th, a Korean morning newspaper also ran an editorial warning that Big Tech must not simply race ahead in competition with one another.
The House will not reconvene until after the November midterm elections. Until then, it will remain undecided who will stop runaway agents and with what authority.
