A 4-month-old boy comes into the clinic with a fever and a stuffy nose. Vyonne Njeri thinks it’s just a cold. Then a yellow box pops up on her computer telling her to check his heart — because his heart rate is elevated. Njeri is a registered clinical officer in Nairobi, Kenya; she sees patients on her own like a nurse practitioner. When she listens with a stethoscope she hears a whoosh — a sign that the infant could have a congenital heart defect. 

“That’s something I would have missed on any other day,” Njeri says of the visit a few months ago. “That child would have just gone home.” She credits an AI tool that double checks her work for helping her. Njeri referred him to a specialist who confirmed the diagnosis and started him on medications; the child may have to undergo surgery.

The tool, called AI Consult, was tested in a randomized trial of nearly 10,000 patient encounters across 16 Kenyan primary clinics operated by Penda Health, and the results were published this summer in Nature Medicine. Half the clinical officers typed their notes into a computer with OpenAI’s GPT-4o checking their electronic notes in the background.

GPT-4o is a large language model, or LLM — the kind of AI that powers chatbots like ChatGPT. They’ve been trained on much of the internet and can generate text in response to instructions.

The other half used a computer for note-taking without the AI tool.

Here’s how the AI program works. It provides three prompts — green, yellow and red, like a traffic light. The AI looks for possible problems in a patient’s care or gaps in the notes. Green means everything is OK. Yellow means “click me” — the AI found a small problem in the notes and offers some guidance. Red pops up if the AI finds a critical concern based on the notes and alerts the clinician, who may need to act promptly.

For Njeri, these interventions offered welcome feedback: “You need to check on that” or “you’re on the right track.”

An independent panel of six Kenyan family physicians graded the notes. They judged that clinicians for whom AI Consult served as what Njeri calls a “second pair of eyes” produced better diagnoses and treatment plans.

And it cost 4 cents per patient.

The study captures both the potential of AI to improve healthcare and also the challenge of proving it.

But although health workers found this AI checkup helpful, the question of patient benefit was left unanswered. The study did not find evidence of improved outcomes.

There was a 23% decrease in treatment failures such as death or unresolved symptoms, but the decrease was not statistically significant because the number of such cases was low to start with.