AI tool aids primary care decisions but not short-term outcomes in trial

A generative AI decision-support system improved parts of clinicians’ work in primary care but did not improve short-term patient outcomes, according to a trial published in Nature Medicine. The findings matter because the study tested an AI tool in routine care with patients, rather than only in simulations or performance exercises.

The University of Birmingham said the pragmatic, cluster-randomized trial included more than 9,600 patients at 16 primary care clinics in Kenya. The work was delivered by University of Birmingham experts with support from the National Institute for Health and Care Research Biomedical Research Center: Birmingham.

Clinicians were randomly assigned to use an electronic medical record system either with or without an embedded tool called AI Consult, according to the university. The system used a large language model to provide real-time diagnostic and treatment suggestions during consultations.

How the tool worked

According to the study team, AI Consult reviewed information entered by clinicians into the medical record and produced suggestions tied to the patient’s context. The recommendations were aligned with Kenyan national clinical guidelines.

The university said the tool also used a green, yellow and red alert system to flag possible concerns. Patients did not see the AI interface, and clinicians kept responsibility for diagnosis, prescribing and referrals.

• The AI reviewed clinician-entered medical record information.
• It generated diagnostic and treatment suggestions.
• It flagged concerns with a color-coded alert system.

Clinical outcomes did not shift

The Nature Medicine paper reported no statistically significant difference in treatment failure within 14 days. The rate was 2.2% among patients seen with AI-supported care and 2.0% among those receiving standard care, according to the University of Birmingham.

The study team also found no evidence of harm, with similar hospitalization and death rates in both groups. The university said serious outcomes such as hospitalization or death are uncommon in primary care, making modest effects difficult to detect without extremely large trials that could involve more than 100,000 patients.

Bilal Mateen, honorary professor of machine learning for health at the University of Birmingham and chief AI officer at PATH, said the trial addressed whether health AI improves patient outcomes. According to the university, he described the safety findings and decision-making improvements as encouraging, while noting that measurable patient benefit is harder to show in everyday primary care.

Documentation and prescribing costs improved

Although short-term outcomes were unchanged, an independent panel of experienced clinicians found better clinical documentation and treatment planning in the AI-supported group, according to the university. The panel was blinded to whether AI had been used.

Patient satisfaction was similar in both trial groups, the study team reported. The university said that suggests AI support did not change patients’ experience of care during consultations.

The researchers also found that overall antibiotic prescribing rates were similar between groups, according to the university. Antibiotic-related costs were lower in the AI-supported group because clinicians made more cost-conscious prescribing choices.

Alastair Denniston, professor of regulatory science and innovation at the University of Birmingham and lead for health data research at the NIHR Biomedical Research Center: Birmingham, said the study showed AI could be added to real clinical workflows without weakening patient trust or clinician autonomy, according to the university.

Richard Riley, professor of biostatistics at the University of Birmingham and a senior author, said robust trials are needed to set realistic expectations for AI in care, according to the university. He also said the findings’ relevance to higher-income health systems, where usual care may already be stronger, still needs evaluation.

This story draws on original reporting from Medical Xpress.