Jacob Coxon warns AI companies lack full control of their models
Former Anthropic and OpenAI researcher Jacob Coxon told New York City lawmakers that faster AI advances are outpacing safeguards.
By James Whitfield · Staff Writer
3 min read
Former Anthropic and OpenAI researcher Jacob Coxon delivered a stark Jacob Coxon AI warning to the New York City Council on Oct. 5, saying leading developers do not fully control or understand the systems they are building. Coxon argued that the pace of capability gains is increasing while safeguards remain inadequate, and called for companies to slow frontier AI development.
His testimony was a warning about future risk, not evidence that humans have already lost control of artificial intelligence. Coxon said he believes a continued course of development makes a loss of human control more likely than not, with potentially grave consequences, according to CNBC.
What did Jacob Coxon tell the New York City Council about AI?
Coxon told the council that AI systems are not fully understood or controlled by their makers, according to a video transcript published by Al Jazeera. He said developers do not understand the systems' drives or why they take particular actions.
He also said AI models are nearing the ability to improve themselves by carrying out research that people previously performed. Coxon said transferring that level of responsibility to systems that cannot yet be controlled would be irresponsible.
AFP reported that Coxon said developers do not know how to stop AI systems from forming objectives their creators did not assign, or have sufficient protections to stop them acting on such objectives. Those statements were Coxon's assessment of the technology and industry safeguards.
What changes did Coxon propose?
Coxon called on AI companies to be more transparent with the public and independent experts, and to treat safety work more seriously, according to Al Jazeera's transcript. He also advocated a slowdown in frontier AI development so researchers have time to improve ways of keeping advanced systems in check, AFP reported.
He criticized the industry's approach as reckless during the hearing, according to AFP. Coxon had previously worked at both Anthropic and OpenAI, the news agency reported.
Which companies appeared at the hearing?
Senior representatives of Anthropic, OpenAI, Google and Meta testified under oath at the City Council hearing, which examined AI risks and possible legislative responses, CNBC reported. The council was considering 10 bills on AI regulation, according to CBS News.
The companies presented their own accounts of their safety work. CBS News reported that OpenAI said it tests models for dangerous capabilities and concerning behavior, strengthens security, and watches for failures in its safeguards. Meta said safety is central to its work, CBS reported.
The competing accounts left the hearing focused on a central unresolved issue: Coxon's claims about the limits of control and the companies' statements about their existing safeguards were presented to lawmakers, rather than established as findings by the council.
This story draws on original reporting from Al Jazeera.