Diagnosis (general) · AI doctors and chatbots
Chatbots ace medical exams, but people using them didn't get better health advice
Oxford researchers gave 1,298 UK adults made-up medical scenarios. On their own, the chatbots named the right condition 94.9% of the time. But people using them did no better than people using Google or the NHS website. Rating: Early research.
How far away is this?
A well-run randomised study, published in Nature Medicine in February 2026, but with pretend scenarios, not real patients, so it's Early research.
Get the weekly How Far
One email every Friday morning (Sydney time) on which medical AI actually reached patients, and what it means for you. Free, in plain English.
Where is it approved?
| Region | Regulator | Status | Notes |
|---|---|---|---|
| United States | FDA | Not applicable | Not a medical product. |
| European Union | CE mark / EMA | Not applicable | Not a medical product. |
| United Kingdom | MHRA | Not applicable | Not a medical product. |
| Australia | TGA | Not applicable | Not a medical product. |
General chatbots aren't approved medical devices. "Register not searched yet" means we haven't searched that regulator's register; it doesn't mean the product is or isn't available. Approval in one country isn't approval in another.
What is it?
This was a randomised study. People were handed one of ten scenarios written by doctors, like a bad headache, and asked what might be wrong and whether to stay home, see a GP, go to A&E or call an ambulance. Some got a chatbot (GPT-4o, Llama 3 or Command R+), the rest used whatever they'd normally use.
What's the evidence?
Tested alone, the chatbots named a relevant condition in 94.9% of cases and picked the right next step in 56.3% on average.[1]
Evidence stage: Early research · #c1
People using the same chatbots named a relevant condition in less than 34.5% of cases and picked the right next step in less than 44.2%, no better than the comparison group. The researchers blame how people and chatbots talk to each other, such as missing details on both sides.[1]
Evidence stage: Early research · #c2
The scenarios were invented, not real illnesses, and the study ran in 2024 with three models (GPT-4o, Llama 3 and Command R+). Newer chatbots may do differently. The authors declare no competing interests.[1]
Evidence stage: Early research · #c3
What does it mean for me?
A chatbot knowing the answer doesn't mean you'll get it out of it. If you use one for symptoms, give it all the details, and if you're unsure how urgent something is, call your GP or a nurse line.
What could I ask my doctor?
- When should I see a GP rather than look something up?
These are general starting points, not a recommendation to choose any particular test or treatment.
If you're worried about your health right now, contact your doctor or local health advice line. In an emergency, contact your local emergency number.
Short videos, under a minute
We make short, plain-English videos about AI in medicine, including what it can't do yet. Follow How Far for the next one.
Change history
- : First published.
General information only, not medical advice. Talk to your doctor about your situation. AI-assisted source check. Not clinically reviewed.
References
- Nature Medicine (2026): Reliability of LLMs as medical assistants for the general public. doi.org (2026). Peer-reviewed randomised trial. https://doi.org/10.1038/s41591-025-04074-y
Publisher shown as the website hosting each source. Dates are given where the source label includes them.