Within three days, warnings about AI risk came three times in the U.S. AI industry. On September 11, news emerged that a bill to guarantee human oversight of AI and stop systems that escape control had been introduced in the U.S. Congress. Around the 12th, Anthropic Chief Executive Officer (CEO) Dario Amodei publicly called on the industry to slow the development race.
In the same week, AI researcher Jacob Coxon left Anthropic and posted a warning about AI risk on X. The debate over who should set the rules for AI flared up again.
It is rare for the head of a company in the thick of the race to talk about putting on the brakes. Amodei posted an essay titled 'We Must Pace the Frontier' on his personal site, darioamodei.com. Here, the frontier refers to the front line where the highest-performing, most advanced AI models compete. In the essay, he said he has worked in the AI field for 12 years.
Some tech company executives pushed back, saying the United States cannot cede leadership in the AI race.
The voices from inside the companies were sharper. Coxon, who has worked at both OpenAI and Anthropic, warned as he left Anthropic, his most recent employer, that Anthropic and its competitors are gambling with people's lives. Whistleblowers also spoke out, saying the rules the industry has set for itself are not enough. Their criticism was aimed at the industry's race as a whole.
Washington's response also picked up speed. When former and current AI researchers in the United States warned that AI could soon lead to human extinction, lawmakers took notice. Rep. Anna Paulina Luna called for an AI 'kill switch.'
A kill switch is a cutoff mechanism built in so that people can immediately shut down a system showing abnormal behavior. The bill reported on the 11th also put human oversight front and center.
The remedies floated in political circles vary widely in severity. Even a proposal to impose a 100% federal tax rate on incentives given to data centers was raised. There were also calls for a complete halt to advanced AI development.
Proposals to create new cutoff standards to block AI agents are also being discussed. AI agents are programs that handle multistep tasks on their own without people giving instructions at every step.
If Anthropic alone slows its pace and competitors do not follow, only the side that slowed down bears the loss. Executives' objection that they cannot cede leadership also draws strength from that anxiety. The path of enforcement by law also has gaps. In an environment where multiple AI agents run simultaneously in different data centers, it must first be decided where to install a kill switch and who has the authority to pull it and when.
Korea's design of AI regulation will also find it hard to sidestep this debate. Provisions that rely on companies' voluntary commitments can lose force the moment competition intensifies. In the United States, the CEO of a company leading the race and a researcher who left that company exposed those limits within a single week.
Deciding whether to leave human oversight as a one-line declaration or to write even the authority and procedures for stopping into the provisions becomes the task at hand. Objections that regulation erodes competitiveness are bound to arise in Korea just the same.
Once AI agents revise schedules and even handle payments on users' behalf, the basis on which users can trust them is closer to a properly working kill switch than to a company's good intentions. The debate that began in Washington over who should be given the authority to activate a kill switch has handed Korea the same homework.
