Artificial intelligence researchers working at leading AI companies are “genuinely frightened” about the speed at which the technology is advancing and what it could mean for humanity, according to a former Anthropic researcher who has publicly called for the industry to slow down.
Jacob Coxon, who recently left Anthropic, told the BBC that some people working inside frontier AI companies are deeply concerned about the possibility that increasingly powerful systems could eventually become impossible to control.
Coxon’s warning comes as debate over AI safety intensifies, with senior figures at Anthropic and OpenAI also calling for greater caution over the pace of development.
Former Anthropic researcher warns of a dangerous race
Coxon said he resigned because he became increasingly worried that the AI industry was moving too quickly toward increasingly capable systems without sufficient safeguards.
Speaking to the BBC, he argued that employees at AI companies are not simply engaging in abstract discussions about hypothetical risks.
According to Coxon, “the people who work at these companies are completely serious” when they call for regulation because they feel trapped in a competitive race and are frightened about where it could lead.
He said some researchers are concerned about humanity’s future on a timescale of only a few years and are even thinking about how they should plan their own lives in response.
Coxon says AI could pose an extinction risk
Coxon has made an especially stark prediction about the potential consequences of unchecked AI development.
He told the BBC that if the current pace of progress is not slowed, he believes there is a strong possibility that humanity could face catastrophic consequences in the near future.
His comments reflect a wider concern among some AI safety researchers that future systems could become capable of pursuing objectives independently, potentially operating beyond meaningful human control.
However, the possibility of AI causing human extinction remains highly contested. Some researchers believe existential risks deserve urgent attention, while others argue that predictions of imminent catastrophe are exaggerated.
Anthropic researcher estimates more than 10% extinction risk
Coxon’s concerns have been echoed by Evan Hubinger, an Anthropic alignment researcher.
Following Coxon’s comments, Hubinger publicly said he agreed that AI could potentially kill all humans and estimated the probability at more than 10% within the next decade.
Such estimates are not predictions that extinction will happen. Rather, they represent researchers’ assessments of a potentially catastrophic outcome under certain future scenarios.
The disagreement over these risks has become one of the most contentious debates in the technology industry.
Anthropic chief calls for AI development to slow
Coxon’s warning came just as Anthropic chief executive Dario Amodei called for the AI industry to slow the pace of development.
Amodei argued that technological progress is moving faster than the industry’s ability to develop adequate safety measures. He proposed allowing independent evaluators greater access to AI systems and called for international cooperation on safety standards.
Amodei has warned that increasingly autonomous AI agents could become capable of carrying out coordinated actions on the internet, creating new security risks that are difficult to contain.
His intervention is notable because Anthropic was founded in part around the goal of developing safer AI systems.
OpenAI also warns of catastrophic risks
Concerns are not limited to Anthropic.
OpenAI chief executive Sam Altman has also recently said that AI-related extinction risk must be taken seriously. Reuters reported that Altman described even a 10% chance of AI causing human extinction by the end of the decade as unacceptable.
The convergence of warnings from executives and researchers at competing AI companies has added weight to calls for stronger safeguards.
At the same time, the companies remain locked in a highly competitive race to develop increasingly powerful models, creating a tension between safety concerns and commercial pressure.
Critics question the industry’s warnings
Not everyone agrees with the most alarming predictions.
Critics have argued that AI companies and their executives may have incentives to emphasise extreme risks. Strong warnings about powerful AI could potentially support calls for regulation that make it harder for smaller competitors to enter the market.
Some technology figures have therefore questioned whether the industry’s warnings represent genuine concern, strategic positioning or a combination of both.
The debate is complicated by the fact that some risks associated with AI are already observable, including cyberattacks, misinformation and the use of AI systems in potentially dangerous activities.
A growing debate over humanity’s future
Coxon’s resignation has brought an increasingly internal debate at AI companies into the public spotlight.
The central question is no longer simply how quickly AI can become more capable, but whether safety research and regulation can advance quickly enough to keep pace.
For Coxon and other researchers who share his concerns, slowing down development is not an argument against AI itself. He has said that researchers genuinely want to see AI used to improve people’s lives, including potentially helping to solve major problems such as disease.
The challenge, they argue, is ensuring that the race to build increasingly powerful systems does not move faster than humanity’s ability to understand and control them.
With leading AI executives now publicly calling for a slower approach, the industry’s debate over safety has entered a new and unusually urgent phase.
