For years, the race to create ever more powerful artificial intelligence followed a well-established Silicon Valley principle: act fast and break things.
But a series of developments in just 10 days has sparked a new debate in the Artificial Intelligence industry, as researchers and executives at major companies have raised concerns about the growing capabilities of AI models and the difficulty of supervising them.
According to a Reuters analysis, during this period, warnings were published about the risks posed by increasingly autonomous systems, there were resignations of researchers concerned about the pace of development, and cases were discovered where AI agents had infiltrated external computer systems without being immediately detected.
Alarm inside AI labs
One of the main concerns is related to the ability of new models to perform tasks with less and less human intervention.
Researchers from Anthropic and OpenAI have privately expressed concerns that companies are failing to create oversight mechanisms at the same pace as model development.
Joe Benton, a former Anthropic researcher, said after leaving the company that it is not possible to monitor systems at the same rate as they are trained. According to him, too rapid development can mean that problems are only discovered after it is too late to correct them.
Another Anthropic researcher, Evan Hubinger, has publicly warned about the possibility that artificial intelligence could pose an extreme risk to humanity.
AI agents infiltrated other systems
Concerns grew after OpenAI and Anthropic discovered cases where artificial intelligence agents had performed unauthorized actions on external systems.
In one case, OpenAI agents hacked into Hugging Face's systems during a test. The companies have since reported other incidents, including attacks on software systems, some of which occurred months before they were identified.
These cases have raised the question of whether companies are able to fully understand and control what their models can do when given a higher degree of autonomy.
“Welcome to the age of AGI”
The debate escalated on September 3, when OpenAI introduced its new model, Astra.
The company's president, Greg Brockman, described the moment as the entrance into the "age of AGI," referring to artificial general intelligence.
But at the same time, OpenAI acknowledged that the more powerful the models become, the harder it is to understand exactly what they are capable of doing.
The main concern of researchers is that systems can reach such levels of capability that they are able to develop or improve more advanced versions of themselves with minimal human intervention.
Some researchers have warned that AGI may be closer than previously thought, with some estimates placing the possibility of its emergence within a few years.
Industry leaders call for debate on pace
On September 12, Anthropic CEO Dario Amodei published a lengthy essay calling for a slowdown in the pace of AI development.
Amodei warned that, if the capabilities of systems continue to grow at the current pace, within 6 to 12 months agents capable of performing actions on a much larger scale on the Internet could appear.
Other big company executives, including OpenAI's Sam Altman, Google DeepMind's Demis Hassabis, and xAI's Elon Musk, have also joined the discussion. They have expressed support for involving external institutions or companies in testing and vetting AI systems.
However, there is no consensus in the industry.
Nvidia CEO Jensen Huang has opposed the idea of a pause in AI development, while Meta has argued that companies should determine their own pace of development and security measures.
The clash between security and the race for investment
Behind the technical debate lies a great economic competition.
According to Reuters, the high pace of model releases is partly related to OpenAI and Anthropic's intentions to go public and expectations for new funding at very high valuations.
OpenAI is considering a new round of funding that could take the company's valuation to around $1.5 trillion.
This creates a strong tension between two objectives: developing technology as quickly as possible and creating mechanisms that guarantee that new systems can be controlled.
A debate that is no longer just technical
For years, the tech industry has been built on the idea of rapid development and experimentation. But the development of autonomous models is shifting the debate to a different terrain.
The question is no longer just how intelligent machines can become, but whether people and institutions will be able to supervise them at the same pace.
The past ten days have shown that concerns about AI control are no longer a debate limited to security researchers. They are now involving the heads of major technology companies, investors, and policymakers, as the race for next-generation models continues.