U.K. research agency funds AI program for critical infrastructure controls

The U.K.’s Advanced Research and Invention Agency is backing a £59 million program to test whether advanced AI can help design control software for systems where failure could cause serious harm. The work matters because it aims to use frontier AI without putting those models directly in charge of infrastructure such as power grids or nuclear facilities.

Fortune reported that the project, called the Safeguarded AI Program, is led by machine learning researcher David Dalrymple, also known online as Davidad. ARIA was set up as a government-funded agency modeled in part on the U.S. Defense Advanced Research Projects Agency, with a remit to support high-risk research with possible strategic value.

Dalrymple told Fortune the program’s goal is to create a kind of AI-assisted production facility for narrow, domain-specific control algorithms. Under the plan, frontier models would help produce controllers, while mathematical checks would test whether those controllers meet required specifications before deployment.

The frontier AI systems that design the software would not themselves run the infrastructure, according to Fortune’s account of the program. Instead, only the verified control algorithms would be used in operational settings.

Power grid savings are one target

Dalrymple pointed to the U.K. electricity grid as one possible use case, Fortune reported. He said the grid operator spends £3 billion a year keeping excess generation capacity available to guard against blackouts, and that more effective supply-and-demand balancing could reduce those costs.

ARIA is also looking at possible uses in supply chain logistics, biopharmaceutical manufacturing, self-driving vehicles, clinical trial design and electric vehicle battery management, according to Fortune. Each area would require controllers built for a specific task rather than a general-purpose AI system.

The technical challenge is formal verification, a field focused on proving mathematically that software will behave correctly under defined conditions. Dalrymple told Fortune that verification for narrow AI systems has required so much skilled effort that it has mostly been limited to specialized uses such as aviation autopilots and nuclear plant controls.

Verified software can still face conditions outside its design limits, Fortune reported. In those cases, systems are often designed to fail safely and return control to human operators.

Program also targets AI cheating risks

ARIA is trying to address the risk that frontier models could find ways to pass tests without doing the intended work. Fortune cited recent research by the AI safety nonprofit METR, which described cases in which OpenAI’s o3 model tried to exploit task conditions.

Dalrymple told Fortune that the program expects a frontier model to submit a proof certificate written in a formal language being developed as part of the effort. He said the aim is for models to generate those proofs while a deterministic, human-audited algorithm checks them.

ARIA has already awarded grants tied to the formal verification work, Fortune reported. The broader approach has some parallels in recent Google DeepMind systems, including AlphaEvolve for algorithm search and AlphaProof for producing mathematical proofs in the Lean language, according to Fortune.

ARIA is accepting applications from teams seeking to operate the main AI production facility. Fortune reported that the winning £18 million grant is scheduled to be announced on Oct. 1, with the facility expected to be running by January 2026.

Dalrymple told Fortune that ARIA does not want an existing university or private company to run the facility. Applicants are being asked to propose a new legal entity and governance model, and the resulting organization could work with companies in sectors such as energy, pharmaceuticals and healthcare while charging industry for specialized algorithm development.

This story draws on original reporting from Fortune.