Anthropic, The AI Safety And Research Company Behind Claude, Warns That Humans Could Lose Control Over AI Systems – AfroTech

Anthropic believes humans could soon lose control over AI altogether.

Anthropic is the AI safety and research company behind Claude, which is valued at nearly a trillion dollars after raising $65 billion in Series H funding led by Altimeter Capital, Dragoneer, Greenoaks, and Sequoia Capital in May, as AFROTECH™ previously reported. The funding will be used for safety and interpretability research, scaling computing capacity, products, and partnerships.

“This funding will help us serve the historic demand we are experiencing, stay at the research frontier, and bring Claude to more of the places where work happens,” said Krishna Rao, chief financial officer of Anthropic, in a press release.

In a new blog post shared by Anthropic, the company says it has been tasking AI systems with a growing share of AI development. While it notes progress is being made at a faster rate, it also shares that this means the technology is inching toward “fully autonomously designing and developing its own successor.”

“This is called recursive self-improvement. We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for,” the blog post said. “Using public benchmarks and previously unreported data from within Anthropic, The Anthropic Institute is showing that AI is already accelerating the development of AI systems. To take just one example: today, Anthropic engineers on average ship 8x as much code per quarter as they did from 2021-2025.”

Anthropic believes the trend toward AI systems that can build their own successors would be a win for science, healthcare,  and other areas globally. It also warns that there would be a greater risk of humans losing control of AI systems.

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” Anthropic said, according to The Straits Times.

“Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures,” Anthropic continued.

It also noted in its blog post that the method for securing, monitoring, and shaping the technology’s behavior will become increasingly important “if systems are capable of fully building their own successors.”