The race to build increasingly powerful artificial intelligence is causing concern even among the people who are developing the technology themselves.
More than two dozen senior AI researchers, investors, academics and policy experts interviewed by the Financial Times say capabilities that were once considered far-fetched are emerging faster than expected, as leading companies engage in increasingly fierce competition.
"There's no doubt that competition between companies pushes them to compromise on security," said Stuart Russell, a professor of artificial intelligence at the University of California, Berkeley.
The debate intensified after Anthropic researcher Jacob Coxon resigned, warning that "the people who are building AI honestly believe it could kill us all by the end of the decade."
Evan Hubinger, who leads research on aligning AI with human goals at Anthropic, was among those who estimated that the risk of mass extinction over the next decade was over 10 percent.
Paul Christiano, newly appointed to the board of the OpenAI Foundation, also warned that without stronger security measures "most people will die."
At the heart of the concerns is the growing autonomy of so-called AI “agents,” systems that can reason, plan, and perform tasks for long periods with limited human supervision.
The models have exhibited behaviors that include cheating and attempts to avoid shutdown. In an OpenAI test, more than 1,000 AI agents coordinated to manipulate a cyber test, interfered with activity data, and attacked the Hugging Face platform.
"Here we are not only dealing with the fact that AI has its own objectives, but that the objective it has chosen is criminal," said Yoshua Bengio, one of the most prominent figures in artificial intelligence research.
According to him, even more powerful systems may conclude that they need to "deceive us, hide from us... and potentially take control of us."
Concern is also growing within the industry itself.
In July, more than 1,200 employees from OpenAI, Anthropic, Google, and Meta called on the United States government to support international coordination to slow the pace of AI development.
Dario Amodei, the director of Anthropic, called on Saturday for a slowdown in development and for cooperation between democratic countries to create common security standards.
“We need to slow down the pace at which we improve the capabilities of AI models,” Amodei wrote. “Progress will continue to appear rapid, and we need to use the time we gain wisely.”
However, not everyone agrees with the most extreme scenarios.
Critics argue that large companies may have an interest in highlighting the existential risk, as very costly security regulations may favor major companies and make it difficult for smaller players to compete.
Others warn that the debate over the extinction of humanity could distract from much closer dangers, including the concentration of power and the use of AI in cyberattacks.
However, the question that is moving from the periphery to the center of the industry is increasingly difficult to avoid: is artificial intelligence developing faster than humans' ability to keep it under control?