In most jobs, when an AI is wrong, you fix the draft. In health, you might not get the chance.
I'm John, a senior AI and full-stack engineer. I spent the last year building a health-information assistant, Tabibu, and I'm starting this series with the question I get asked most: why is health so much harder than the other places I have shipped AI?
1. A mistake costs more
A wrong dose is not a typo. In a normal product, a bad answer costs a retry. In health, someone may act on it. So I stopped asking "is the answer good?" and started asking "what is the worst believable wrong answer here?" For Tabibu the answer was a dose, so the system does not tell anyone to start, stop or change a medicine.
2. The AI sounds equally sure whether it is right or wrong
This is the one that surprised me as an engineer. A model gives a calm, fluent answer when it knows and when it is guessing. A patient usually cannot tell the difference, because telling the difference is the expert's job.
So I stopped trusting tone as a signal. Answers from Tabibu show their sources, and when the guidelines do not cover a question it says so and labels the answer as general information. A confident voice is not a reason to trust anything.
3. The rules are strict for good reasons
Where software influences a medical decision, regulators in many places treat it as a medical device, and that takes evidence. I do not make that claim for Tabibu: it is an information tool, it has no clinician review yet, and the limits are listed at tabibu.health/trust.
For a builder, the lesson is to design the product so its boundaries are written down and testable, not implied.
Is it hopeless?
No. A 2026 Mount Sinai study in Communications Medicine tested 20 AI models on more than 10 million responses. Adding a short safety reminder cut potentially harmful clinical choices from 16.6% to 10.1%, with reductions in 19 of the 20 models. I read Mount Sinai's press release, not the paper itself. Its lead researcher said a reminder is one safeguard, not a substitute for clinical oversight. That is the shape I trust: safety work helps, and it does not reach zero.
ECRI's 2026 patient safety report ranks AI in clinical diagnosis first on its top-10 list of concerns, which fits what I have seen: the question is not whether AI is good or bad for health, but where it is safe enough, and who checks.
What I'd ask you
What is one health decision you would never leave to an AI? Over the next few weeks I'll cover who builds these tools, how they make money, who sets the rules, and where it fails.
I post short videos in this series on LinkedIn and TikTok; come say hi.
General information, not medical advice.
Top comments (3)
To answer your question: telling someone they're fine.
I work on a mental health screening app, and the worst believable wrong answer in that space isn't an alarming result; it's a reassuring one. An alarming result usually leads to a conversation with a professional, which is a safe outcome even when the alarm was wrong. A calm "nothing to worry about" can be the reason someone never makes that call. So the decision I'd never hand to a model is the one that closes the door.
Your point 2 is why. A model sounds just as sure when it's guessing, and reassurance is exactly where people want to believe it.
Publishing the limits on a trust page is a good habit, by the way. "No clinician review yet" is a sentence most health apps leave out.
Thanks, Ahmed, and for the follow. I framed my answer around a dose, but a reassurance that stops someone from making the call is the same failure with no number attached. That is a better version of the question than the one I asked.
Tabibu isn't a screening tool and I wouldn't present it as one, so I'm curious how you handle it where you are: for a borderline result, does the app always point to a human option, or only past a threshold?
Band-based, and your question made me look at the edge of it.
The advice line comes from a fixed table per questionnaire, not from the model. On the PHQ-9, "consider speaking to a doctor or counselor" starts at the moderate band (10 and up). Below that, the advice is about sleep and routine. Every PDF report says it isn't a diagnosis and points to a professional, and an answer that sounds like risk opens a separate support screen with crisis lines, whatever the total.
The weak spot is exactly the one you're pointing at: a 9 and a 10 are one point apart and get very different messages. I think a score near a cut-off should say so ("close to the next band") and keep the human option on screen. That's going on our list.