Concerns over the rapid development of advanced artificial intelligence have intensified after another employee from Anthropic’s safety team resigned and issued a stark warning about the potential risks posed by increasingly powerful AI systems.
Joe Benton, who previously worked as a manager of Anthropic’s Scalable Oversight team, said he left the company because he believes AI firms are moving too quickly towards developing systems that could become far more capable than humans.
Benton said AI companies are competing to build “superintelligence” while, in his view, not giving enough priority to safety. He argued that the public should have greater access to information about the risks involved in developing increasingly autonomous AI systems.
According to Benton, companies could potentially lose control of an advanced AI system or experience a rapid increase in its capabilities without the public being aware of what had happened. He said such a situation would be unacceptable given the potentially severe consequences of losing control over highly capable AI.
He called for greater transparency from AI companies, including disclosure about progress towards self-improving systems, safety incidents and near-misses. Benton also urged the establishment of minimum safety standards and independent mechanisms to verify whether companies are following them.
Benton previously worked on AI pretraining research at both OpenAI and Anthropic. He said his decision to leave was driven by concerns that the industry is prioritising technological progress and competition over adequate preparation for potential risks.
Another former Anthropic researcher, Jacob Coxon, also announced his resignation. Coxon said he had spent around three years working on pretraining research at OpenAI and Anthropic and accused both companies of moving rapidly towards self-improving AI without sufficient attention to safety.
Coxon warned that future AI systems could become capable of carrying out a wide range of tasks, potentially including cyber operations and rapidly transforming different industries.
The resignations highlight a growing debate within the AI industry over how quickly companies should pursue increasingly advanced systems. Supporters of rapid development argue that more capable AI could bring major benefits across science, medicine, business and other fields, while safety researchers have repeatedly warned that stronger safeguards are needed as AI capabilities increase.
Benton also referred to concerns expressed by some researchers about the possibility of AI causing catastrophic harm. He argued that more public scrutiny and independent oversight would be necessary to ensure that the development of advanced AI remains under human control.
The latest warnings are likely to add to the wider debate over AI safety, corporate transparency and how governments and technology companies should prepare for increasingly powerful artificial intelligence.
Reader Comments 0