Over two days, on September 15 and 16, voices warning of the risks of AI came one after another. Yoshua Bengio, known as a 'godfather of AI,' warned that humanity could lose the power to control AI, and Anthropic Chief Executive Officer (CEO) Dario Amodei called on the industry to respond to safety together. There are counterarguments too. Nvidia CEO Jensen Huang took a position against new regulation, saying AI can be developed quickly even without new safety laws.
As tech industry CEOs have issued warnings calling for regulation one after another, discussion of AI regulation is steadily growing. The issue has also shifted. The positions of Bengio, Amodei and Huang are divided over who controls development and through what mechanisms.
Amodei's prescription goes a step further. In an essay titled 'We Must Pace the Frontier,' he argued that the industry should slow the pace of frontier model development until safeguards, oversight systems and alignment techniques catch up. Frontier models refer to the highest-performing of the latest AI models each company releases.
Alignment is the work of refining AI so that it acts in line with the goals and values people intend. The essay was published last weekend.
Amodei put forward both a request for companies to organize safety efforts together and a request to rein in the development of the most advanced models.
Bengio's warning targets something more fundamental. Going beyond the level of a particular product malfunctioning, it warns that the very power of people to stop and fix AI could disappear. Behind the repeated calls from those favoring safeguards to rein in development lies this concern about loss of control.
Jensen Huang makes his case from the opposite side. His argument is that AI development can continue quickly without creating new regulation. Huang also said that companies can decide for themselves whether a product is safe enough to release. Following Huang's logic, the party that verifies safety and the party that decides on release become the same company.
The mood in Washington is closer to Huang's argument.
President Trump has been reluctant toward calls to slow AI development or strengthen regulation. Unless the government applies the brakes, mechanisms to control AI have no choice but to rely on voluntary industry commitments or companies' internal standards.
Both solutions have gaps. Amodei-style slowing works only if the entire industry keeps the same pace, and if even one company pushes ahead, the commitment loses its force. Under Huang-style autonomy, release standards are set inside the company, so there is little room to examine from outside whether those standards were sufficient. The loss of control Bengio spoke of will not resolve itself under either approach.
The debate is moving beyond the stage of declarations and into institutional design. The next points of contention are who decides when to slow development, what authority to give oversight bodies, and whether there are means to externally verify the results of safety checks that companies have graded themselves. Agreement is not easy because the two companies stand in different places to begin with, with Anthropic building models and Nvidia supplying chips to the AI industry.
The debate between Amodei and Huang divides over whether the authority to stop AI when it tries to do things it was not assigned should be kept only inside companies or also given to outside oversight bodies.
