Between Trust and Skepticism: How Can We Coexist With AI Doctors?

In 2025, if you wake up late at night with a scratchy, dry throat and a slight fever, your first instinct may no longer be to rummage through the medicine cabinet for fever reducers. Instead, you might reach for your phone, open an AI health app, and describe your symptoms. Seconds later, a response appears: “Preliminary assessment: early-stage wind-heat cold. Recommended: drink more water, take Yin Qiao detox tablets, and avoid spicy food.”

AI-powered medical Q&A is spreading fast. It’s no longer limited to standalone health apps—similar features are now being embedded into general-purpose large models and everyday platforms. Many people use them to “check” tongue coating, facial color, or other visible cues to estimate their condition. On social media, posts sharing AI diagnosis screenshots and self-recovery experiences have become common. With every moment an AI “gets the symptoms right,” trust quietly accumulates.

But the conversation has also taken a darker turn. There have been reports of AI-generated “prescriptions” leading to serious harm, including cases where people developed rare conditions after following misguided advice. One widely discussed overseas incident involved a user who followed health tips generated through ChatGPT conversations and ended up with bromide poisoning—an illness rarely seen today.

So what are the real advantages and risks of using AI for medical guidance, and how should users think about it rationally?

The sudden rise of the “AI doctor”

A wave of AI consultation tools has surged into the market. Most major online pharmacy platforms now include built-in “AI doctor” functions, and many people rely on them to decide what medicine to buy. According to Alibaba Cloud’s 2025 AI Application White Paper for the Healthcare Industry, 101 medical AI models have already been registered in China.

Unlike expensive AI imaging equipment—often costing millions and typically purchased only by top-tier hospitals—these AI “doctors” have an extremely low barrier to entry. No specialized devices. No appointment. Available 24/7. And often free.

They’re especially effective for common illnesses with abundant historical data and familiar symptom patterns—colds, stomach discomfort, headaches, mild inflammation—where they can quickly match guideline-based over-the-counter options or lifestyle interventions.

Many users come away impressed. It feels accurate. It’s fast. And it doesn’t require waiting in long hospital queues. Over time, some people find themselves visiting clinics less often.

For young adults with irregular routines, busy office workers discouraged by overcrowded hospitals, or residents in areas with weaker primary care access, the convenience can be genuinely helpful. In some situations, AI does fill gaps in basic health consultation and reduces anxiety around “what should I do first?”

The risks are growing—quietly, but steadily

Yet behind the convenience, risks can grow without warning. News reports have surfaced of people delaying treatment because they trusted AI guidance too much. In more extreme cases, users have harmed themselves by following flawed recommendations.

One 2025 case drew major attention: a 60-year-old overseas user, obsessed with “clean health,” decided to remove chlorine from his diet entirely. After multiple conversations with ChatGPT, he accepted a suggestion and used bromide compounds as a long-term substitute for table salt. His blood bromide level reportedly reached 1,700 mg/L, more than 200 times normal.

He later developed symptoms such as hallucinations, unstable gait, and memory confusion, and was diagnosed with chronic bromide poisoning. This condition was once seen in the early 20th century due to medication misuse, and has largely disappeared with modern medical standards—yet it resurfaced because of AI-generated “wellness advice.”

Around the same time, a cautionary case appeared closer to home. A woman (pseudonym: Ms. Liao Xinhua) experienced persistent dry coughing for months and repeatedly used an AI consultation platform to self-diagnose. The AI produced shifting conclusions such as “allergic bronchitis” and “gastroesophageal reflux irritation,” and suggested medications like anti-allergy drugs and acid suppressors. She bought and took them herself, felt temporary relief, stopped, relapsed, and fell into a cycle of “medicate, improve, stop, relapse.”

For eight months, she did not visit a hospital. Only after significant weight loss and blood-streaked sputum did she seek medical care. A chest CT showed multiple thick-walled cavities and patchy infiltrates in both lungs. Doctors strongly suspected secondary pulmonary tuberculosis in an active stage. The physician reportedly told her that if she had come two months earlier, her condition likely would not have progressed this far.

These cases force a hard question: if AI seems so capable, why can it produce advice that is dangerous—or even absurd? Is it just occasional error, or something deeper in how AI “medical thinking” actually works?

Can you trust an AI-generated prescription?

When someone types “sore throat, low fever, fatigue” and instantly receives advice like “drink water and take Yin Qiao,” it’s easy to feel that this isn’t so different from a real doctor’s recommendation. But the logic underneath is very different—and far less reliable than it looks.

Medical decision-making begins with comprehensive, objective assessment—often summarized by the classic clinical process of observing, listening/smelling, questioning, and physical examination. Today’s AI can interpret text and some images, but it cannot truly examine a patient.

It can’t reliably see whether you look pale or short of breath. It can’t hear subtle changes in breathing. It can’t feel swollen lymph nodes. And it can’t pick up what a trained clinician might notice from your tone, hesitation, anxiety, or missing history. It depends entirely on what you type—and that input can be incomplete, subjective, or simply wrong.

Many people also struggle to describe medical details accurately. They may not distinguish dry cough from irritation-induced choking cough, or dull pain from cramping pain. Once key details are distorted, the output can easily drift. Even small issues—like a blurry photo or poor lighting—can skew image-based judgments.

Then there’s the long-tail problem. Large models handle common conditions better because they’re trained on high-frequency patterns. But many serious diseases start with symptoms that look ordinary. Early tuberculosis may present as dry cough and night sweats; lupus may begin with low fever and joint pain. When the probability is low but the risk is high, AI can mistakenly categorize the situation as a routine cold or mild infection—potentially delaying the best treatment window.

Most fundamentally, language models can “hallucinate.” They do not inherently verify truth; they predict what a plausible answer looks like. That means an AI can confidently invent nonexistent medications, cite guidelines that don’t exist, or recommend harmful “health hacks” that any medical student would flag immediately.

And if harm occurs, accountability is often unclear. Many platforms include disclaimers stating that outputs are “for reference only” and “do not constitute medical practice.” In practice, this can leave users carrying most of the consequences—even when the AI experience was presented as authoritative.

So while AI prescriptions may feel fast and professional, they can hide multiple layers of risk: distorted input, blind spots in disease recognition, model hallucinations, and unresolved responsibility.

How the industry is trying to fix the problem

To be fair, the healthcare AI space is not standing still. Many teams are actively exploring safer designs.

At the source level, training materials are being restricted to high-confidence medical databases. Some platforms are attempting to limit AI outputs to validated sources such as authoritative journals and clinical guidelines, reducing the chance of misinformation.

At the product level, the role of AI is shifting. Instead of directly delivering diagnoses or medication plans to patients, AI is increasingly positioned as a backstage assistant—helping organize symptoms into structured notes, summarizing patient histories, and supporting doctors with documentation and triage.

At the scope level, the “generalist fantasy” is giving way to specialized tools. Early products tried to be all-purpose family doctors and ended up shallow. Now, more developers focus on narrow areas such as dermatology image recognition, blood glucose management for diabetes, or retinal screening in ophthalmology. With better task boundaries and higher-quality data, performance improves—and regulatory approval becomes more realistic.

Overall, the most responsible path is clear: effective medical AI must stay anchored within the human clinical decision framework, supporting professionals rather than replacing them.

Even major AI platforms are becoming more cautious

As AI consultation features spread across apps, more people have formed the habit of “asking AI first” when they feel unwell. Yet globally, many tech companies and regulators are moving toward more cautious boundaries.

OpenAI, for example, has explicitly limited ChatGPT from providing specific diagnoses or medication instructions. If a user asks, “I’ve had headaches for three days—what medicine should I take?”, the system typically responds that it cannot provide medical advice and encourages consulting qualified professionals.

That caution reflects a core reality: most general-purpose models are essentially making probabilistic guesses based on public text patterns. They are not clinically trained, not medically validated, and do not hold any license to practice. Their responses imitate how humans might answer—not what your condition truly is.

Using AI wisely: where it helps, and where it becomes dangerous

AI consultation has real value—but only in the right scenarios.

In long-term, high-frequency, low-risk settings, AI can be genuinely useful: health monitoring, medication reminders, follow-up check-ins, rehabilitation tracking, and lifestyle coaching—especially when a diagnosis has already been confirmed by a clinician. In these cases, AI is not the decision-maker. It’s the organizer and executor.

But when it comes to new symptoms, unclear causes, screening for urgent or severe illness, or situations requiring individualized judgment, the risk rises sharply. Convenience can hide danger. A casual “it’s probably just heatiness” may cause someone to ignore early warning signs of something much more serious. A seemingly reasonable self-care plan can delay proper evaluation for tuberculosis, autoimmune disease, or other high-stakes conditions.

If you’ve already been diagnosed by a top-tier hospital and are managing recovery or chronic disease, using AI for tracking and reminders can be safe and helpful—with one rule: it must not cross into prescribing, replace follow-ups, or create the illusion that you “don’t need to go back to the doctor.”

When your body sends unusual signals, the safest and most responsible choice is still the same: go to a hospital and see a real clinician.

Because medicine is not only science—it is also deeply human. It requires empathy to understand fear, experience to catch subtle clues beyond textbooks, and ethics to weigh treatment, risk, and dignity. Those are qualities today’s medical AI tools still cannot replicate.