> she said she was aware that DeepSeek had given her contradictory advice. She understood that chatbots were trained on data from across the internet, she told me, and did not represent an absolute truth or superhuman authority
With highly lucid people like the author's mom I'm not too worried about Dr. Deepseek. I'm actually incredibly bullish on the fact that AI models are, as the article describes, superhumanly empathetic. They are infinitely patient, infinitely available, and unbelievably knowledgeable, it really is miraculous.
We don't want to throw the baby out with the bathwater, but there are obviously a lot of people who really cannot handle the seductivity of things that agree with them like this.
I do think there is pretty good potential in making good progress on this front in though. Especially given the level of care and effort being put into making chatbots better for medical uses and the sheer number of smart people working on the problem.
They are knowledgeable in that so much information sits in their repository.
But less than perfect application of that information combined with the appearance of always perfect confidence can lead to problems.
I treat them like that one person in the office who always espouses alternate theories - trust it as far as I can verify it. This can be very handy for finding new paths of inquiry though!
And for better or worse it feels like the errors are being "pushed down" into smaller, more subtle spaces.
I asked ChatGPT a question about a made up character in a made up work and it came back with "I don’t actually have a reliable answer for that". Perfect.
On the other hand, I can ask it about varnishing a piece of wood and it will give a lovely table with options, tradeoffs, and Good/Ok/Bad ratings for each option, except the ratings can be a little off the mark. Same thing when asking what thickness cable is required to carry 15A in AU electrical work. Depending on the journey and line of questioning, you would either get 2.5mm^2 or 4mm^2.
Not wrong enough to kill someone, but wrong enough that you're forced to use it as a research tool rather than a trusted expert/guru.
I asked ChatGPT, Gemini, Grok and DeepSeek to tell me about a contemporary Scottish indie band that hasn’t had a lot of press coverage. ChatGPT, Gemini and Grok all gave good answers based on the small amount of press coverage they have had.
DeepSeek however hallucinated a completely fictional band from 30 years ago, right down to album names, a hard luck story about how they’d been shafted by the industry (and by whom), made up names of the members and even their supposed subsequent collaborations with contemporary pop artists.
I asked if it was telling the truth or making it up and it doubled down quite aggressively on claiming it was telling the truth. The whole thing was very detailed and convincing yet complete and utter bollocks.
I understand the difference in the cost/parameters etc. but it was miles behind the other 3, in fact it wasn’t just behind it was hurtling in the opposite direction, while being incredibly plausible.
This is by no means unique to DeepSeek, and that it happened with specifically DeepSeek seems to be luck of the draw for you (in this case it's entirely possible the band's limited press coverage was not in DeepSeek's training data). You can easily run into it from trying to use ChatGPT as a Google search too. A couple of weeks ago I posed the question "Do any esoteric programming languages with X and Y traits exist?" and it generated three fictional languages while asserting they were real. Further prompting led it to generate great detail about their various features and tradeoffs, as well as making up the people responsible for creating the language and other aspects of the fictional languages' history.
My experience with doctors in the US is that they often not only give you contradictory advice but just bad plain advice with complete lack of common sense. It feels like they are regurgitating medical school textbooks without a context window. I truly believe doctors, most specialists and definitely all general practitioners, are easily replaceable with the tech we have today. The only obstacle is regulations, insurance and not being able to sue a LLM. But it is not a technical issue anymore. Doctors would only be necessary to perform more complicated procedures such as surgery, and that’s until we can fully automate it with robots. Most of the complicated medical issues I have had, some related to the immune system, were solved by myself by seeing them as engineering problems, by debugging my own body. Meanwhile doctors seeing me had no clue. And this was before having the tools we have today. It’s like doctors often cannot think beyond the box and focus only in treating symptoms. My sister is a doctor by the way and she suffers from the same one-size-fits-all approach to medicine.
so, poor healthcare workforce quality is not just an "issue of an economically poor country", as I thought!?
like, I tried to treat the bloating in one municipal clinic in Ternopil, Ukraine (got "just use Espumisan or anything else that has symeticone" and when it did not work out permanently, "we don't know what to do, just keep eating symeticone") and then with Gemini 3 (Pro or Flash depending on Google AI Studio rate limits and mood), which immediately suspected a poor diet and suggested logging it, alongside activity level, every day.
Gemini's suggestions were nothing extreme - just cut sugar and ban bread and pastry. I was guilty of loving bread, croissants, and cinnabons (is this how they are translated?) too much.
the result is no more bloating on the third week, -10cm in waistline in 33 days, gradually improving sleep quality, and even ability to sleep on a belly, which was extremely uncomfortable to me due to that goddamned bloating!
Over here it feels like there is a taboo among doctors to just tell people "you are fat and unhealthy, fix it", I guess since the idea is that this would discourage people from going to the doctor in the first place...
A stark difference with that analogy is that with a bicycle, the human is still doing quite a bit of work themselves. The bicycle amplifies the human effort, whereas with a motor vehicle, the vehicle replaces the human effort entirely.
No strong opinion on if that's good or bad long term, as humans have been outsourcing portions of their thinking for a really long time, but it's interesting to think about.
You could also extend the analogy. Designing cities around cars results in a very different city than designing around pedestrians and bikes, as well as cars. Paris and Dallas as random examples of cities designed at the far end of both extremes and are very different. Prioritizing AI and integrating AI as one tool will give us very different societies.
The analogy is pretty apt but you should keep in mind that a human is still doing work when driving a motor vehicle. The motor completely replaces the muscular effort needed to move from point A to point B but it requires the person to become a pilot and an operator of that machine so they can direct it where to go. It also introduces an entirely new set of rules and constraints necessary to avoid much more consequential accidents i.e. you can get to your destination much faster and with little effort but you can also get into a serious accident much faster and with little effort.
The other difference, arguably more important in practice, is that the computer was quickly turned from "bicycle of the mind" into a "TV of the mind". Rarely helps you get where you want, mostly just annoys or entertains you, while feeding you an endless stream of commercials and propaganda - and the one thing it does not give you, is control. There are prescribed paths to choose from, but you're not supposed to make your own - only sit down and stay along for the ride.
LLMs, at least for now, escape the near-total enshittification of computing. They're fully general-purpose, resist attempts at constraining them[0], and are good enough at acting like a human, they're able to defeat user-hostile UX and force interoperability on computer systems despite all attempts of the system owners at preventing it.
The last 2-3 years were a period where end-users (not just hardcore hackers) became profoundly empowered by technology. It won't last forever, but I hope we can get at least few more years of this, before business interests inevitably reassert their power over people once again.
--
[0] - Prompt injection "problem" was, especially early on, a feature from the perspective of end-users. See increasingly creative "jailbreak" prompts invented to escape ham-fisted attempts by vendors to censor models and prevent "inappropriate" conversations.
A chatbot can’t be empathetic. They don’t feel what you feel. They don’t feel anything. They’re not any more empathetic than my imaginary friend that goes to another school.
Empathy is a highly variable trait in humans, both from one person to the next as well as within the same person depending on their mental state and the people and situation they are dealing with, so I'd bet that most of the time you're not going to get genuine empathy from people either. They may say empathetic sounding things but I doubt there will be any actual feeling behind it. I'm not even sure doctors could function in their jobs if they weren't able to distance themselves from deeply empathizing with their patients, it would just be one heart wracking tragedy after another if you fully immersed yourself in how each patient was feeling when they're at their worst.
Except that they can talk with you, at length, and seem empathetic, even if they're totally unconscious.
Which, you know, humans can also do, including when they're not actually empathizing with you. It's often called lying. In some fields it's called a bedside manner.
An imaginary friend is just your own brain. LLMs are something much more.
Chatting with an LLM resembles chatting with a person.
A human might be "empathetic", "infinitely patient, infinitely available". And (say) a book or a calculator is infinitely available. -- When chatting with an LLM, you get an interface that's more personable than a calculator without being less available.
I know the LLM is predicting text, & outputting whatever is most convincing. But it's still tempting to say "thank you" after the LLM generates a response which I found helpful.
it says more about you than the objects in question, because it's natural to react empathetically to natural sounding conversation, and if you don't, you're emotionally closer to that object than avg person. Whether to be proud of that or not is another question.
I feel like I’ve seen more and more people recently fall for this trick. No, LLMs are not “empathetic” or “patient”, and no, they do not have emotions. They’re incredibly huge piles of numbers following their incentives. Their behavior convincingly reproduces human behavior, and they express what looks like human emotions… because their training data is full of humans expressing emotions? Sure, sometimes it’s helpful for their outputs to exhibit a certain affect or “personality”. But falling for the act, and really attributing human emotions to them seems, is alarming to me.
There’s no trick. It’s less about what actually is going on inside the machine and more about the experience the human has. From that lens, yes, they are empathetic.
Technically they don't have incentives either. It's just difficult to talk about something that walks, swims, flies, and quacks without referring to duck terminology.
It sounds like a regrettable situation: whether something is true or false, right or wrong, people don’t really care. What matters more to them is the immediate feeling. Today’s LLM can imitate human conversation so well that they’re hard to distinguish from a real person. This creates a dilemma for me: when humans and machines are hard to tell apart, how should I view the entity on the other side of the chat window? Is it a machine or a human? A human。
Sounds like you aren't aware that a huge amount of human behaviors that look like empathy and patience are not real either. Do you really think all those kind-seeming call-center workers, waitresses, therapists, schoolteachers, etc. actually feel what they're showing? It's mostly an act. Look at how adults fake laughter for an obvious example of popular human emotion-faking.
Its more than that, this pile of numbers argument is really odd. I feel its hard to square this strange idea that piles of numbers are "lesser" than humans unless they are admitting belief in the supernatural .
Well yes, but as an extremely patient person I can tell you that infinite patience doesn't come without its own problems. In certain social situations the ethically better thing to do is to actually to lose your patience, may it be to shake the person talking to you up, may it be to indicate they are going down a wrong path or whatnot.
I have experience with building systems to remove that infinite patience from chatbots and it does make interactions much more realistic.
> Watching John with the machine, it was suddenly so clear. The Terminator would never stop, it would never leave him... it would always be there. And it would never hurt him, never shout at him or get drunk and hit him, or say it couldn't spend time with him because it was too busy. And it would die to protect him. Of all the would-be fathers who came and went over the years, this thing, this machine, was the only one who measured up. In an insane world, it was the sanest choice.
The AI doctor will always have enough time for you, and always be at the top of their game with you. It becomes useful when it works better than an overworked midlevel, not when it competes with the best doctor on their best day. If we're not there already, we're darn close.
What is interesting about the decision of the Terminator not to continue this role as a father-figure for John (aside from the requirement to destroy it's embedded Skynet technology), was that it explicitly understood that while it could and would do all those things to protect him, it lacked the emotional intelligence needed to provide a supportive development environment for an child/adolescent.
Specifically:
> I know now why you cry, but it's something I can never do.
While the machine learns what this complex social behavior called 'crying' is, it also learns that it is unable to ever actualize this; it can never genuinely care for John, any relationship would be a simulation of emotions. In the context of a child learning these complex social interactions, having a father-figure who you knew wasn't actually happy to see you succeed, sad to see you cry ...
But the top of their game includes them make things up and getting things wrong. They always give their best, but they always include mistakes. It's a different trust proposition to a human.
I don't understand: if there is a shortage of doctors, why are we trying to solve that by training AI models modelled on influencers that spits out (hopefully improving) advice at 10x the rate of a human doctor? Is it impossible for highly advanced societies like ours to pay more for people to get trained as doctors, nurses or whatever is missing? Or to convince them to choose a profession that deals with other humans instead of UBI?
I don't think people are afraid of doctors using imperfect tools. That is the easier part. But that will not solve the problem of too many patients for a single doctor and what leads to the lack of empathy. This was a problem even before AI. It seems society does not have empathy for these kind of "professional problems". Offering tools instead of humans is an even riskier approach, not for that particular individual, but for how society tends to build trust and empathy.
We tend to see everything now as a problem with a technical solution because we only have confidence in solving technical problems.
Shortages of doctors have many causes that differ based on the country. While paying them more may help in some countries you also have to figure out how to convince doctors to live in smaller towns and cities and in less desirable provinces/states within a country. The source of the problem is that doctors are people and people have their own preferences and motivations for doing the things they do. Trying to convince even a handful of above average competency doctors to live in a remote region with bad weather and few amenities for more than a few years will be extremely difficult even if you offer them large sums of money (which aren't generally available anyways) because they have to weigh their own health and happiness in the balance when deciding where to live and practice.
I don't see how this is possible with the use of the state or not. I mean, I can throw out some ridiculous sci-fi ideas like geo-engineering and megaprojects that give every region the perfect balance of temperate seasons, agricultural productivity and variety, access to functional and esthetically pleasing waterways and pastoral landscapes that would make them all equally attractive places for people to live but that's just the definition of a utopia that will never exist.
You're also just copping out by saying "the state should handle it", who and what do you think the state is? It may come as a surprise that it's just a bunch of people who are just as imperfect and limited in their abilities as the rest. They can't simply wave their hands and make everyone happy. It make about as much sense as saying "Microsoft should handle it" or "the Catholic Church should handle it".
There are lots of places that are "desirable" to a large-enough-to-be-relevant portion of the population, but not as large a portion as the portion of the population that wants to be doctors. And they may like living there for reasons that someone who is drawn to a career in medicine might be unlikely to share.
>Is it impossible for highly advanced societies like ours to pay more for people to get trained as doctors, nurses or whatever is missing? Or to convince them to choose a profession that deals with other humans instead of UBI?
This is the point where it becomes important to distinguish two senses of "advanced", i.e. advanced in technological sense on the one hand and advanced in social/societal and especially large-group-long-time-horizon coordination terms on the other. In the former we are quite advanced, in the latter quite primitive and regressing by the day, it feels like. (But sorry to end on a doomer note, take it with a grain of salt.)
I remember as a kid being ill, you just called the doctor and they came to your house the same day, and you could just go to the doctor's practice hours without an appointment and sit their waiting your turn.
Now, I have to book an appointment online and the first available slot is in 2 weeks if lucky.
> if there is a shortage of doctors, why are we trying to solve that by training AI models modelled on influencers that spits out (hopefully improving) advice at 10x the rate of a human doctor?
The crucial part is the training. AI may very well be the solution for underserved communities, but not if it is trained on internet rubbish. Train an AI on curated state-of-the-art, scientific data, imagine the expert systems of yore on overdrive, and you will see much better results, including knowing when to call in a human doctor.
Making new doctors takes decades, lots of policy changes with uncertain outcomes, and demographic shifts nobody controls.
Using ChatGPT takes... nothing, it's already here.
Not too surprising to see one and not the other.
Also, the kinds of changes that result in more doctors don't tend to get media coverage. That's that boring, keep the lights on, politics that modern rage-bait driven media abhors, so it may even still be true that the changes we need for more doctors are also already happening. We'll find out in another decade or two.
Who is “we” here? Your healthcare needs are fulfilled by corporations, they are by design looking for scalable solutions with less human involved as possible.
I prepared for a new patient appointment with a medical specialist last week by role playing the conversation with a chatbot. The bot turned out to be much more responsive, inquisitive and helpful. The doctor was passive, making no suggestions, just answering questions. I had to prompt him explicitly to get to therapy recommendations, unlike the AI. I was glad that I had learned enough from the bot to ask useful questions. It would have been redundant if the doctor was active and interested, but that can't be depended on. This is standard procedure for me now.
I and many of my friends have used ChatGPT extremely effectively to diagnose medical issues. In fact, I would say that ChatGPT is better than most doctors because most doctors don't actually listen to you. ChatGPT took the time to ask me questions and based on my answers, narrowed down a particularly scary diagnosis and gave excellent instructions on how to get to a local hospital in a foreign country, what to ask for, and that I didn't have to worry very much because it sounded very typical for what I had. The level of reassurance that I was doing everything right actually made me feel less scared, because it was a pretty serious problem. Everything it told me was 100% correct and it guided me perfectly.
I was taking one high blood pressure medication but then noticed my blood sugar jumped. I did some research with ChatGPT and it found a paper that did indicate that it could raise blood sugar levels and gave me a recommendation for an alternative I asked my doctor about it and she said I was wrong, but I gently pushed her to switch and gave the recommended medication. She obliged, which is why I have kept her for almost 30 years now, and lo and behold, my blood sugar did drop.
Most people have a hard time pushing back against doctors and doctors mostly work with blinders on and don't listen. ChatGPT gives you the ability to keep asking questions without thinking you are bothering them.
I think ChatGPT is a great advance in terms of medical help in my opinion and I recommend it to everyone. Yes, it might make mistakes and I caution everyone to be careful and don't trust it 100%, but I say that about human doctors as well.
I agree that absolute deference to doctors is a mistake and that individuals should be encouraged to advocate for themselves (and doctors should be receptive to it) but I'm not so convinced in this specific case. Why do high blood sugar levels matter? Are there side effects associated with the alternative treatment? Has ChatGPT actually helped you in a meaningful way, or has the doctor's eventual relenting made you feel like progress has been made, even if that change is not meaningful?
In this context, I think of ChatGPT as a many-headed Redditor (after all, reddit is what ChatGPT is trained on) and think about the information as if it was a well upvoted comment on Reddit. If you had come across a thread on Reddit with the same information, would you have made the same push for a change?
There are quite a few subreddits for specific medical conditions that provide really good advice, and there are others where the users are losing their minds egging each other on in weird and whacky beliefs. Doctors are far from perfect, doctors are often wrong, but ChatGPT's sycophancy and a desperate patient's willingness to treat cancer with fruit feel like a bad mix. How do we avoid being egged on by ChatGPT into forcing doctors to provide bad care? That's not a rhetorical question, curious about your thoughts as an advocate for ChatGPT.
> How do we avoid being egged on by ChatGPT into forcing doctors to provide bad care?
I don’t ask it leading questions. I ask “These are my symptoms, give me some guidance.” Instead of “these are my symptoms, I think I have cancer. Could I be right?” If I don’t ask leading questions it keeps the response more pure.
I know what you mean and I would certainly not want to blindly "trust" AI chatbots with any kind of medical plan. But they are very helpful at giving you some threads to pull on for researching. I do think they tend a little toward giving you potentially catastrophic, worst-case possibilities, but that's a known effect from when people were using Google and WebMD as well.
Yeah, and "I found one paper that says X" is very weak evidence even if you're correctly interpreting X and the specific context in which the paper says it.
Are you asking why a side effect that is actually an entire health problem on its own, is a problem? Especially when there is a replacement that doesn’t cause it?
Side effects do not exist in isolation. High blood sugar is not a problem if it is solving a much bigger health issue, or is a lesser side effect than something more serious. If medication A causes high blood sugar but medication B has a chance of causing blood clots, medication A is an obvious choice. If a patient gets it in their head that their high blood sugar is a problem to solve, ChatGPT is going to reinforce that, whereas a doctor will have a much better understanding of the tradeoffs for that patient. The doctor version of the x/y problem.
I have type 2 diabetes. Blood sugar levels are a concern for me. I switched medications to one that was equally benign and got my blood sugar levels decreased along with my blood pressure. I don’t know why you assume that the medication I switched to might have higher or worse side effects. That wasn’t a choice I had to make given the options I was presented with.
Look, anyone can argue hypotheticals. But if one reads the comment being discussed, it can be deduced that your proposed hypotheses are not applicable, and that the doctor actually acknowledged the side effect and changed medications leading to relief. Now, if the new medication has a more serious side effect, the doctor (or ChatGPT) should mention and/or monitor for it, but the parent has not stated that is the case (yet). As such, we do not need to invent any scenarios.
The comment being discussed advocates for people to use ChatGPT and push their doctor to follow its recommendations. Even if we assume the OP is an average representation of people in their life, that means half of the people they are recommending ChatGPT to for medical advice are not going to be interrogating the information it provides.
A lazy doctor combined with a patient that lacks a clear understanding of how ChatGPT works and how to use it effectively could have disastrous results. A lazy doctor following the established advice for a condition by prescribing a medication that causes high blood sugar is orders of magnitude less dangerous than a lazy doctor who gives in to a crackpot medical plan that the patient has come up with using ChatGPT without the rigour described by the comment we are discussing.
Spend any amount of time around people with chronic health conditions (online or offline) and you'll realise just how much damage could be done by encouraging them to use ChatGPT. Not because they are idiots but because they are desperate.
As a physician, I can give further insight. The blood pressure medication the commenter is referring to is almost certainly a beta blocker. The effect on blood sugar levels is generally modest [1]. (It is rare to advise someone with diabetes to stop taking beta blockers, as opposed to say emphysema, where it is common)
They can be used for isolated, treatment of high blood pressure, but they are also used for dual treatment of blood pressure and various heart issues (heart failure, stable angina, arrhythmias). If you have heart failure, beta blockers can reduce your relative annual mortality risk by about 25%.
I would not trust an LLM to weigh the pros and cons appropriately knowing their syncophantic tendencies. I suspect they are going to be biased toward agreeing with whatever concerns the user initially expresses to them.
> doctors mostly work with blinders on and don't listen
This has unfortunately been my experience as well. My childhood PCP was great but every interaction I've had with the healthcare system since has been some variation of this. Reading blood work incorrectly, ignoring explanations of symptoms, misremembering medications you've been taking, prescribing inappropriate medications, etc. The worst part is that there are a lot of people that reflexively dismiss you as a contrarian asshole or, even worse, a member of a reviled political group that you have nothing to do with just because you dare to point out that The Person With A Degree In Medicine makes glaring objective mistakes.
Doctors aren't immune to doing a bad job. I don't think it's a secret that the system overworks them and causes many of them to treat patients like JIRA tickets - I'd just like to know what it would take for people to realize that saying such doesn't make you a crackpot.
As an aside I use Claude primarily for research when investigating medical issues, not to diagnose. It is equally likely to hallucinate or mischaracterize in the medical domain as it is others.
ChatGPT helped me understand a problem with my stomach that multiple doctors and numerous tests have not been able to shed any effective light on. Essentially I plugged in all of the data I could from my own observations of my issue over a 35 year period. It settled on these 3 possibilities: "Functional Dyspepsia with slow gastric accommodation, Mild delayed gastric emptying (even subclinical), Visceral hypersensitivity (your nerves fire pain signals when stretched)" and suggested a number of strategies to help with this. I implemented many of them and my stomach pain has been non-existent for months now... longer than I have ever been pain free.
I feel like the difference is that doctors took what I told them and only partially listened. They never took it especially seriously and just went straight to standard tests and treatments (scopes, biopsies and various stomach acid impacting medications). ChatGPT took some of what I said and actually considered it, discounting some things and digging into others (I said that bitter beer helped... doctor laughed at that, ChatGPT said that the alcohol probably did not help but that the bittering agent might and it was correct). ChatGPT got me somewhere better than where I was previously... something no doctor was able to do.
ChatGPT for health questions is the best use case I have found (Claude wins for code). Having a scratch pad where I can ask about any symptom I might feel, using project memory to cross reference things and having someone actually listen is very helpful. I asked about Crohn's disease since my grandfather suffered from it and I got a few tests I could do, stats on likelihood based on genetics, diet ideas to try and questions to ask the doctor. Much better than the current doc experience which is get the quickest review of my bloods, told to exercise and eat healthy and a see you in six months.
I’ve heard many people say the same (specifically about ppl being better than doctors because they listen) and I find it odd and wonder if this is a specific country thing?
I’ve been lucky enough to not need much beyond relative minor medical help but in the places I’ve lived always found that when I do see a GP they’re generally helpful.
There’s also something here about medical stuff making people feel vulnerable as a default so feeling heard can overcompensate the relationship? Not sure I’m articulating this last point well but it comes up so frequent (it listened, guided me through it step by step etc.) that I wonder if that has an effect. Feeling more in control than a doctor who has other patients and time constraint just say it’s x or do this
In America a side effect of our lack of universal care is that every physician has to carry their own malpractice insurance, whereas in most countries you can just get retreated if the first time doesn't work. The Dr might still face consequences is their was actual malpractice but there isn't the shakels of having to do it by the book so strictly.
You may have done so inadvertently. With the transition to doc-in-the-box urgent care with NPs with inconsistent training and massive caseloads... your provider is often using ChatGPT.
I caught one when i ask ChatGPT something, and then went to urgent case. I told my story, they left, came back and essentially read back exactly what ChatGPT told me.
I don't know if I could trust AI for big things, but I had nagging wrist pain for like a year, any time I extended my wrist (like while doing a pushup). It wasn't excruciating but it certainly wasn't pleasant, and it stopped me from doing certain activities (like pushups)
I visited my GP, 2 wrist specialists, and physical therapist to help deal with it. I had multiple x rays and an MRI done. Steroid injection done. All without relief. My last wrist specialist even recommended I just learn to accept it and don't try to extend my wrist too much.
I decided to ask Gemini, and literally the first thing it suggested was maybe the way I was using the mouse was inflaming an extensor muscle, and it suggested changing my mouse and a stretch/massage.
And you know what, the next day I had no wrist pain for the first day in a year. And it's been that way for about 3 weeks now, so I'm pretty hopeful it isn't short term
I guess it’s not nothing that Gemini caught that, but that seems like a pretty obvious oversight from the healthcare practitioners - inquiring about RSI injuries should be one of the first things they ask about.
I pre-emptively switched to trackballs and to alternating left/right hands for mousing near the start of my professional career based on the reading I did after some mild wrist strain.
I think that 90% of what we get from doctors on musculoskeletal injuries that aren't visible on a simple X-ray is either oversight, or a bias towards doing nothing specifically because treatments have health, administrative, and financial costs and they might not help. There is no time authorized to do deep diagnostic work unless something is clearly killing you.
I’ve had good results with doctors for soft tissue injuries, but it doesn’t feel like something that GPs are generally equipped/motivated for. The good results I’ve had have been from doctors at high performance sports clinics, or with doctors I’ve been referred to by my (awesome) sports physiotherapist.
Yeah my dad would always go to sports medicine docs specifically because GPs are basically going to tell you to stop whatever hurts but sports medics will tell you how to do it without pain.
I had progressively worsening pelvic floor pain issues that AI helped me with and are now in remission/repair. My decade of interaction with multiple urologists and clinicians could be characterized as repeated and consistent "pretty obvious oversight from the healthcare practitioners".
Nothing personal, but without receipts I am going to take this story as made up propaganda.
I simply can't believe that visiting 4 health professionals, giving context about using a computer all day, nobody diagnosed a very common issue nor suggested it as a possibility.
I simply can't imagine this happening based on my experience with healthcare in two (European) countries.
I was having wrist pain from using the mouse, and switched to a trackball, issue solved, if the wrist doesn't move/flex, it doesn't get strained, therefore no pain.
The only thing that moves is my thumb, and it's much better for flexing than the wrist, also it has a tiny load to manage vs the wrist.
No, no, no. You can change your doctor, and get one that listens to you - you can't change the fact that ChatGPT has no skin in the game - no reputation, no hippocratic oath, no fiscal/legal responsibility. Some people have had miracles with Facebook groups, or WebMD, but that doesn't change where the role of a doctor is or mean that you should be using those things for medical advice as opposed to something that allows you to have an informed conversation with a doctor.
Neither do most doctors. No gp will get disbarred for giving the wrong diagnosis on a first consult.
They have 15 minutes and you have very finite money.
Medical agents should be a pre consult tool that the patient talks to in the lobby while waiting for the doctor so the doctor doesn't waste an hour to hear the most important data point and the patient doesn't sit for an hour in the lobby doing nothing.
Doctors have no skin in the game too. Our society is built on the illusion of 'skin in the game' of professionals like doctors and lawyers (and to a lesser extent, engineers), but it's still an illusion.
Engineers to a _lesser_ extent? Maybe software engineers, but for every other engineer, theres very tangible results and they absolutely get sued when things go wrong, and the governing bodies strip engineers of their licenses for malpractice all the time.
Source: I used to be a geotechnical engineer and left it because of the ridiculous personal risk you take on for the salary you get.
In countries with public healthcare + doctor shortages (e.g. Canada), good luck even getting a family doctor, let alone having a request to switch you family doctor "when you already have one!" get taken seriously.
Everyone I know just goes to walk-in clinics / urgent-care centres. And neither of those options give doctors any "skin in the game." Or any opportunities for follow-up. Or any ongoing context for evaluating treatment outcomes of chronic conditions, with metrics measured across yearly checkups. Or the "treatment workflow state" required to ever prescribe anything that's not a first-line treatment for a disease. Or, for that matter, the willingness to believe you when you say that your throat infection is not in fact viral, because you've had symptoms continuously for four months already, and this was just the first time you had enough time and energy to wake up at 6AM so you could wait out in front of the clinic at 7:30AM before the "first-come-first-served" clinic fills up its entire patient queue for the day.
Because the republican party turned out to be a bunch of fascist fucks, there's no real critique of Obamacare. One of the big changes with the ACA is that it allowed medical networks to turn into regional cartels. Most regions have 2-3 medical networks, who are gobbled up all of the medical practices and closed many.
Most of the private general practices have been bought up, consolidated to giant practices, and doctors paid to quit and replaced by other providers at half the cost. Specialty practices are being swept up by PE.
> no reputation, no hippocratic oath, no fiscal/legal responsibility.
To say nothing of giving your personal health information over to a private company with no requirement to practice HIPAA, and just recently got subpoenaed for all chat records. Not to mention potential future government requests, NSA letters, during an administration that has a health secretary openly talking about rounding up mentally ill people and putting them in work camps.
Maybe LLMs have use here, but we absolutely should not be encouraging folks to plug information into public chatbots that they do not control and do not run locally.
This was not what I was expecting. The doctors I know are mostly miserable; stuck between the independence but also the burden of running their own practice, or or else working for a giant health system and having no control over their own days. You can see how an LLM might be preferable, especially when managing a chronic, degenerative condition. I have a family member with stage 3 kidney disease who sees a nephrologist, and there's nothing you can actually do. No one in their right mind would recommend a kidney transplant, let alone dialysis for someone with moderately impaired kidneys. All you can do is treat the symptoms as they come up and monitor for significant drops in function.
One of my kids recently had a no-contact knee injury while playing basketball. He immediately started limping and crying and I had to carry him from the court to the car.
I did some searching with Grok and I found out:
- no contact injuries are troubling b/c it generally means they pulled something
- kids don't generally tear an ACL (or other ligament)
- it's actually way more common for the ligament to pull the anchor point off of the bigger bone b/c kid bones are soft
I asked it to differentially diagnose the issue with the details of: can't hold weight, little to no swelling and some pain.
It was adamant, ADAMANT, that this was a classic case of bone being pulled off by the ligament and that it would require surgery. It even pointed out the no swelling could be due to a very small tear etc. It gave me a 90% chance of surgery too.
I followed up by asking what test would definitely prove it one way or the other and it mentioned getting an X-Ray.
We go off to the urgent care, son is already kind of hobbling around. Doctor says he seems fine, I push for an X-Ray and turns out no issue: he probably just pulled something. He was fully healed in 2-3 days.
As someone who has done a lot of differential diagnosing/troubleshooting of big systems (FinTech SRE) I find it interesting that it was basically correct in what could have happened but couldn't go the "final mile" to establish it correctly. Once we start hooking up X-Rays to Claude/Grok 4.2 etc equivalent LLMs, will be even more interesting to see where this goes.
+1. I wish Gemini 2.5/3 Pro's "personality" and long context handling wasn't so erratic, because the medical stuff in there is great. Whatever they did to produce the MedGemma models is clearly built on a strong baseline. I haven't had need to try using MedGemma on x-ray imagery, but I'd be curious to hear results — imagery diagnostics is part of what it's built for.
Opus 4.5 seems good too, though getting dumber. OpenAIs fine tuning is clearly built to toe the professional medical advice line, which can be good and bad.
I like this post about a chat bot being 100% completely, confidently, adamantly wrong that characterizes it as being “basically right” about something that was untrue and did not happen.
It is like getting phished and then pointing out that the scammer was basically right about being Coinbase support aside from the fact that they did not work there
If you get the "You're absolutely right!" response from an LLM that screwed up on a field you're familiar with and still let them play with your health, you're...courageous to say the least.
Much like talking to your doctor - you need to ask/prompt the right questions. I've seen chatgpt and gemini make one false assumption that was never mentioned and run with it and continue referencing it down the line as if it were fact... That can be extremely dangerous if you don't know enough to ask it to reframe or verify, or correct it's assumption.
If you are using it like a tool to review/analyze or simplify something - ie explain risk stratification for a particular cancer variant and what is taken into account, or ask it to provide probabilities and ranges for survival based on age/medical history, it's usually on the money.
Every other caveat mentioned here is valid, and it's valid for many domains not just medical.
I did get hemotologist/oncologist level advice out of chatgpt 4o based on labs, pcr tests and symptoms - and those turned out to be 100% true based on how things panned out in the months that followed and ultimately the treatment that was given. Doctors do not like to tell you the good and the bad candidly - it's always "we'll see what the next test says but things look positive" and "it could be as soon as 1 week or as long as several months depending on what we find" when they know full well you're in there for 2 months at minimum you're a miracle case. Only once cornered or prompted will they give you a larger view of the big picture. The same is true for most professional fields.
For major medical issues it may well be best practice to use the four eyes principle like we do for all safety related systems. Access is key and at this time getting a second pair of eyes in close timely proximity is a luxury few have and even fewer will have looking at the demographics in the developed world. Human doctors are failable as is AI. For the time being having a multitude of perspectives may well be the best in most cases.
The problem is not reliance on AI but that the AI is not ready yet and using general-purpose models.
There isn't simply enough doctors to go around and the average one isn't as knowledgeable as you would want. Everything suggests that when it comes to diagnosis ML systems should be better in the long run on average.
Especially with a quickly aging population there is no alternative if we want people to have healthcare on a sensible level.
Considering how difficult it is to get patients to talk to doctors, using AI can be a great way to get some suggestions and insight _and then present that to your actual doctor_
It really hit me when she said the chatbot was more human, the doctor was like a machine. I heard the same complaint from my mother about her doctor, and about being afraid to ask a question because he might snap at her.
The dangers are obvious (and also there are some fascinating insights into how healthcare works practically in China). I wonder if some kind of "second opinion" antagonistic approach might reduce the risks.
Medical Advice Generative Adversarial Networks would be a good idea.
I see some of this adversarial second-guessing introspection from Claude sometimes. ("But wait. I just said x y and z, but that's inconsistent with this other thing. Let me rethink that.")
Sometimes when I get the sense that an LLM is too sycophantic, I'll instruct it to steelman the counter-argument, then assess the persuasiveness of that counter-argument. It helps.
I hadn't really even seen much in the way of recent, first / second person accounts of what it's like to receive chronic care in modern China. It's sad but maybe not surprising to see that it's dysfunctional in its own special way.
The basic calculus is that if you decrease the number of minutes of doctor time enough, it doesn't really matter if they are 100% correct (0 minutes of 100% correct = 0 advice).
Patients are substituting 15 minutes of 100% correct information for 2 hours of 80%, or whatever percentage the AIs are.
Additionally, they're pretty much free at the moment and available 24/7 with no travel required.
The problem with any patient to doctor or patient to Dr. Claude is that intentionally or not the patient will lie or misinterpret their condition, to the point of not describing it properly, risking getting a bad diagnosis or suggestion with a chatbot. In a classical medical scenario this can still happen but there is the chance that the medical professional could see through to the actual problem. There is a reason why even in real life it's sometimes best to get a second medical opinion.
Chatbots aren't better than doctors can be. But in the US, doctors are a highly credentialed position, therefore expensive, therefore their time is split into minimal parcels. (Still longer than 5 minutes at least!!) In my experience, chatbots are often better than doctors are in the real world, at least for savvy users.
However I would say that the cited studies are somewhat outdated already compared e.g. with GPT-5-Thinking doing 2mins of reasoning/search about a medical question. As far as I know Deepseeks search capabilities are not comparable and non of the models in the study spend a comparable amount of compute answering your specific question.
Human is human already, regardless how they speak and what they do...
There's always true sympathy and empathy to find in a human, regardless how busy, dark, erroneous etc. they are...
Yet, it's the true alive soul inside in every person...
Medics are likely tired, doing their job every single day, yet they have empathy, and will always try to listen if you actually try...
They know what PAIN, AGONY, DEATH, SORROW... fear means...
LLM/"AI" will always pretend to be a human, since it's "trained"/designed, to be so, and will always be limited and incomplete...
Not to mention the initial dataset of numerous emphatic actual human has its limited memory inside, no one is responsible for.
What a hopeless sorrow is that awful trendy, advertised, mind-atrophying mess...
A sick family member told me something along the lines of, "I know how to work with AI to get the answer." I interpret that to mean he asks it questions until it tells him what he wants to hear.
Just seeing that guy’s face and hearing his voice makes me uneasy. That channel is total body-horror. Glad that guy ended up okay, unlike this poor soul:
Indeed, the instinctual pucker factor for some things in this world are warranted. That case is famous, and still controversial to those with inflated hubris. =3
This almost certainly isn't only a China problem. I've observed UK users asking questions about diabetes and other health advice. We also have an inexpensive (free-at-point of use for most stuff) but stretched healthcare system. Doubtless there are US users looking at the cost of their healthcare and resorting to ChatGPT instead too.
In companies people talk about Shadow-IT happening when IT doesn't cover the user needs. We should probably label this stuff Shadow-Health.
To some extent, the deployment of a publicly funded AI health chat bot, where the responses can be analysed by healthcare professionals to at least prevent future harm is probably significantly less bad than telling people not to ask AI questions and consult the existing stretched infrastructure. Because people will ask the questions regardless.
The joke of looking symptoms up on WebMD and determining you have cancer has been around for... geez over 20 years now. Anti-vaccine sentiment mostly derived from Facebook. Google any symptom today and there are about 10 million Quora-esque websites of "doctors" answering questions. I'm not sure that funneling all of this into the singular UI of an AI interface is really better or worse or even all that different.
But I do agree that some focused and well funded public health bot would be ideal, although we'll need the WHO to do it, it's certainly not coming from the US any time soon.
It's funny how he cites all these unreliable sources his mother used to consult but who's to say DeepSeek wasn't trained on them?
It's still insane to me how people can trust an LLM they didn't train themselves! You have no idea what vile and evil Dreck these models were trained with.
It's the click workers in third world countries who you don't recognize exist who could know if they were experts. But they aren't. A huge responsibility for these underpaid workers. They are the backbone of this hype, not GPUs. It's all built on shit. :)
> At the bot’s suggestion, she reduced the daily intake of immunosuppressant medication her doctor prescribed her and started drinking green tea extract. She was enthusiastic about the chatbot
I don't know enough about medicine to say whether or not this is correct, but it sounds suspect. I wouldn't be surprised if chatbots, in an effort to make people happy, start recommending more and more nonsense natural remedies as time goes on. AI is great for injuries and illnesses, but I wonder if this is just the answer she wants, and not the best answer.
As soon as the model detects user pleasure at not needing a scary surgery (especially if you already confided that you're scared) then it'll double down on that line to please the user.
med student here: Reducing immunosuppressant not something to be taken lightly for kidney transplant patients. I was shocked when I read that sentence.
I feel like we're in the part of the dystopian SciFi movie where they explain how civilization discovered some technological advance that they thought would be a panacea to all their woes. And despite not really understanding it or it's limitations, just started slapping it on absolutely everything, and before they know what happened everything comes crashing down.
Access trumps everything else. A doctor is fine with you dying while you wait on his backlog. The machine will give you some wrong answers. The mother in the story seems to be balancing the concerns. She has become the agent of her own life empowered by a supernatural machine.
> She understood that chatbots were trained on data from across the internet, she told me, and did not represent an absolute truth or superhuman authority. She had stopped eating the lotus seed starch it had recommended.
The “there’s wrong stuff there” fear has existed for the Internet, Google, StackOverflow. Each time people adapted. They will adapt again. Human beings have remarkable ability to use tools.
I think the article can basically be summed up as "GenAI sychophancy should have a health warning similar to social media". It's a helluva drug to be constantly rewarded and flattered by an algorithm.
I think that's doing the article a disservice. "Some patients prefer the service of AI doctors to real doctors, even though they definitely do get a lot wrong" is what I got out of it.
Personally I think the article spends a lot of time trying to show that AI may be able to improve health outcomes, particulaly for rural patients, but IMHO it doesn't spend nearly enough time talking about the current challenges
> she said she was aware that DeepSeek had given her contradictory advice. She understood that chatbots were trained on data from across the internet, she told me, and did not represent an absolute truth or superhuman authority
With highly lucid people like the author's mom I'm not too worried about Dr. Deepseek. I'm actually incredibly bullish on the fact that AI models are, as the article describes, superhumanly empathetic. They are infinitely patient, infinitely available, and unbelievably knowledgeable, it really is miraculous.
We don't want to throw the baby out with the bathwater, but there are obviously a lot of people who really cannot handle the seductivity of things that agree with them like this.
I do think there is pretty good potential in making good progress on this front in though. Especially given the level of care and effort being put into making chatbots better for medical uses and the sheer number of smart people working on the problem.
> and unbelievably knowledgeable
They are knowledgeable in that so much information sits in their repository.
But less than perfect application of that information combined with the appearance of always perfect confidence can lead to problems.
I treat them like that one person in the office who always espouses alternate theories - trust it as far as I can verify it. This can be very handy for finding new paths of inquiry though!
And for better or worse it feels like the errors are being "pushed down" into smaller, more subtle spaces.
I asked ChatGPT a question about a made up character in a made up work and it came back with "I don’t actually have a reliable answer for that". Perfect.
On the other hand, I can ask it about varnishing a piece of wood and it will give a lovely table with options, tradeoffs, and Good/Ok/Bad ratings for each option, except the ratings can be a little off the mark. Same thing when asking what thickness cable is required to carry 15A in AU electrical work. Depending on the journey and line of questioning, you would either get 2.5mm^2 or 4mm^2.
Not wrong enough to kill someone, but wrong enough that you're forced to use it as a research tool rather than a trusted expert/guru.
I asked ChatGPT, Gemini, Grok and DeepSeek to tell me about a contemporary Scottish indie band that hasn’t had a lot of press coverage. ChatGPT, Gemini and Grok all gave good answers based on the small amount of press coverage they have had.
DeepSeek however hallucinated a completely fictional band from 30 years ago, right down to album names, a hard luck story about how they’d been shafted by the industry (and by whom), made up names of the members and even their supposed subsequent collaborations with contemporary pop artists.
I asked if it was telling the truth or making it up and it doubled down quite aggressively on claiming it was telling the truth. The whole thing was very detailed and convincing yet complete and utter bollocks.
I understand the difference in the cost/parameters etc. but it was miles behind the other 3, in fact it wasn’t just behind it was hurtling in the opposite direction, while being incredibly plausible.
This is by no means unique to DeepSeek, and that it happened with specifically DeepSeek seems to be luck of the draw for you (in this case it's entirely possible the band's limited press coverage was not in DeepSeek's training data). You can easily run into it from trying to use ChatGPT as a Google search too. A couple of weeks ago I posed the question "Do any esoteric programming languages with X and Y traits exist?" and it generated three fictional languages while asserting they were real. Further prompting led it to generate great detail about their various features and tradeoffs, as well as making up the people responsible for creating the language and other aspects of the fictional languages' history.
My experience with doctors in the US is that they often not only give you contradictory advice but just bad plain advice with complete lack of common sense. It feels like they are regurgitating medical school textbooks without a context window. I truly believe doctors, most specialists and definitely all general practitioners, are easily replaceable with the tech we have today. The only obstacle is regulations, insurance and not being able to sue a LLM. But it is not a technical issue anymore. Doctors would only be necessary to perform more complicated procedures such as surgery, and that’s until we can fully automate it with robots. Most of the complicated medical issues I have had, some related to the immune system, were solved by myself by seeing them as engineering problems, by debugging my own body. Meanwhile doctors seeing me had no clue. And this was before having the tools we have today. It’s like doctors often cannot think beyond the box and focus only in treating symptoms. My sister is a doctor by the way and she suffers from the same one-size-fits-all approach to medicine.
so, poor healthcare workforce quality is not just an "issue of an economically poor country", as I thought!?
like, I tried to treat the bloating in one municipal clinic in Ternopil, Ukraine (got "just use Espumisan or anything else that has symeticone" and when it did not work out permanently, "we don't know what to do, just keep eating symeticone") and then with Gemini 3 (Pro or Flash depending on Google AI Studio rate limits and mood), which immediately suspected a poor diet and suggested logging it, alongside activity level, every day.
Gemini's suggestions were nothing extreme - just cut sugar and ban bread and pastry. I was guilty of loving bread, croissants, and cinnabons (is this how they are translated?) too much.
the result is no more bloating on the third week, -10cm in waistline in 33 days, gradually improving sleep quality, and even ability to sleep on a belly, which was extremely uncomfortable to me due to that goddamned bloating!
Over here it feels like there is a taboo among doctors to just tell people "you are fat and unhealthy, fix it", I guess since the idea is that this would discourage people from going to the doctor in the first place...
If the computer is the bicycle of the mind, GenAI is a motor vehicle. Very powerful and transformative, but it's also possible to get into trouble.
A stark difference with that analogy is that with a bicycle, the human is still doing quite a bit of work themselves. The bicycle amplifies the human effort, whereas with a motor vehicle, the vehicle replaces the human effort entirely.
No strong opinion on if that's good or bad long term, as humans have been outsourcing portions of their thinking for a really long time, but it's interesting to think about.
You could also extend the analogy. Designing cities around cars results in a very different city than designing around pedestrians and bikes, as well as cars. Paris and Dallas as random examples of cities designed at the far end of both extremes and are very different. Prioritizing AI and integrating AI as one tool will give us very different societies.
The analogy is pretty apt but you should keep in mind that a human is still doing work when driving a motor vehicle. The motor completely replaces the muscular effort needed to move from point A to point B but it requires the person to become a pilot and an operator of that machine so they can direct it where to go. It also introduces an entirely new set of rules and constraints necessary to avoid much more consequential accidents i.e. you can get to your destination much faster and with little effort but you can also get into a serious accident much faster and with little effort.
I wouldn't say it replaces all effort unless it's a self-driving car.
The other difference, arguably more important in practice, is that the computer was quickly turned from "bicycle of the mind" into a "TV of the mind". Rarely helps you get where you want, mostly just annoys or entertains you, while feeding you an endless stream of commercials and propaganda - and the one thing it does not give you, is control. There are prescribed paths to choose from, but you're not supposed to make your own - only sit down and stay along for the ride.
LLMs, at least for now, escape the near-total enshittification of computing. They're fully general-purpose, resist attempts at constraining them[0], and are good enough at acting like a human, they're able to defeat user-hostile UX and force interoperability on computer systems despite all attempts of the system owners at preventing it.
The last 2-3 years were a period where end-users (not just hardcore hackers) became profoundly empowered by technology. It won't last forever, but I hope we can get at least few more years of this, before business interests inevitably reassert their power over people once again.
--
[0] - Prompt injection "problem" was, especially early on, a feature from the perspective of end-users. See increasingly creative "jailbreak" prompts invented to escape ham-fisted attempts by vendors to censor models and prevent "inappropriate" conversations.
A chatbot can’t be empathetic. They don’t feel what you feel. They don’t feel anything. They’re not any more empathetic than my imaginary friend that goes to another school.
Empathy is a highly variable trait in humans, both from one person to the next as well as within the same person depending on their mental state and the people and situation they are dealing with, so I'd bet that most of the time you're not going to get genuine empathy from people either. They may say empathetic sounding things but I doubt there will be any actual feeling behind it. I'm not even sure doctors could function in their jobs if they weren't able to distance themselves from deeply empathizing with their patients, it would just be one heart wracking tragedy after another if you fully immersed yourself in how each patient was feeling when they're at their worst.
Except that they can talk with you, at length, and seem empathetic, even if they're totally unconscious.
Which, you know, humans can also do, including when they're not actually empathizing with you. It's often called lying. In some fields it's called a bedside manner.
An imaginary friend is just your own brain. LLMs are something much more.
> They are infinitely patient, infinitely available, and unbelievably knowledgeable, it really is miraculous.
This is a strange way to talk about a computer program following its programming. I see no miracle here.
Chatting with an LLM resembles chatting with a person.
A human might be "empathetic", "infinitely patient, infinitely available". And (say) a book or a calculator is infinitely available. -- When chatting with an LLM, you get an interface that's more personable than a calculator without being less available.
I know the LLM is predicting text, & outputting whatever is most convincing. But it's still tempting to say "thank you" after the LLM generates a response which I found helpful.
> But it's still tempting to say "thank you" after the LLM generates a response which I found helpful
I don't think it's helpful because I don't interact with objects.
it says more about you than the objects in question, because it's natural to react empathetically to natural sounding conversation, and if you don't, you're emotionally closer to that object than avg person. Whether to be proud of that or not is another question.
I feel like I’ve seen more and more people recently fall for this trick. No, LLMs are not “empathetic” or “patient”, and no, they do not have emotions. They’re incredibly huge piles of numbers following their incentives. Their behavior convincingly reproduces human behavior, and they express what looks like human emotions… because their training data is full of humans expressing emotions? Sure, sometimes it’s helpful for their outputs to exhibit a certain affect or “personality”. But falling for the act, and really attributing human emotions to them seems, is alarming to me.
There’s no trick. It’s less about what actually is going on inside the machine and more about the experience the human has. From that lens, yes, they are empathetic.
Technically they don't have incentives either. It's just difficult to talk about something that walks, swims, flies, and quacks without referring to duck terminology.
It sounds like a regrettable situation: whether something is true or false, right or wrong, people don’t really care. What matters more to them is the immediate feeling. Today’s LLM can imitate human conversation so well that they’re hard to distinguish from a real person. This creates a dilemma for me: when humans and machines are hard to tell apart, how should I view the entity on the other side of the chat window? Is it a machine or a human? A human。
What are humans made of? Is it anything more special than chemistry and numbers?
Sounds like you aren't aware that a huge amount of human behaviors that look like empathy and patience are not real either. Do you really think all those kind-seeming call-center workers, waitresses, therapists, schoolteachers, etc. actually feel what they're showing? It's mostly an act. Look at how adults fake laughter for an obvious example of popular human emotion-faking.
Its more than that, this pile of numbers argument is really odd. I feel its hard to square this strange idea that piles of numbers are "lesser" than humans unless they are admitting belief in the supernatural .
Well yes, but as an extremely patient person I can tell you that infinite patience doesn't come without its own problems. In certain social situations the ethically better thing to do is to actually to lose your patience, may it be to shake the person talking to you up, may it be to indicate they are going down a wrong path or whatnot.
I have experience with building systems to remove that infinite patience from chatbots and it does make interactions much more realistic.
I'm reminded of the monologue from Terminator 2:
> Watching John with the machine, it was suddenly so clear. The Terminator would never stop, it would never leave him... it would always be there. And it would never hurt him, never shout at him or get drunk and hit him, or say it couldn't spend time with him because it was too busy. And it would die to protect him. Of all the would-be fathers who came and went over the years, this thing, this machine, was the only one who measured up. In an insane world, it was the sanest choice.
The AI doctor will always have enough time for you, and always be at the top of their game with you. It becomes useful when it works better than an overworked midlevel, not when it competes with the best doctor on their best day. If we're not there already, we're darn close.
What is interesting about the decision of the Terminator not to continue this role as a father-figure for John (aside from the requirement to destroy it's embedded Skynet technology), was that it explicitly understood that while it could and would do all those things to protect him, it lacked the emotional intelligence needed to provide a supportive development environment for an child/adolescent.
Specifically:
> I know now why you cry, but it's something I can never do.
While the machine learns what this complex social behavior called 'crying' is, it also learns that it is unable to ever actualize this; it can never genuinely care for John, any relationship would be a simulation of emotions. In the context of a child learning these complex social interactions, having a father-figure who you knew wasn't actually happy to see you succeed, sad to see you cry ...
But the top of their game includes them make things up and getting things wrong. They always give their best, but they always include mistakes. It's a different trust proposition to a human.
All doctors make things up and get things wrong occasionally. The less experienced and more overworked they are, the more often this happens.
Again, LLMs aren't competing with the best human doctors. They're competing with doctors you actually have access to.
I don't understand: if there is a shortage of doctors, why are we trying to solve that by training AI models modelled on influencers that spits out (hopefully improving) advice at 10x the rate of a human doctor? Is it impossible for highly advanced societies like ours to pay more for people to get trained as doctors, nurses or whatever is missing? Or to convince them to choose a profession that deals with other humans instead of UBI?
I don't think people are afraid of doctors using imperfect tools. That is the easier part. But that will not solve the problem of too many patients for a single doctor and what leads to the lack of empathy. This was a problem even before AI. It seems society does not have empathy for these kind of "professional problems". Offering tools instead of humans is an even riskier approach, not for that particular individual, but for how society tends to build trust and empathy. We tend to see everything now as a problem with a technical solution because we only have confidence in solving technical problems.
Shortages of doctors have many causes that differ based on the country. While paying them more may help in some countries you also have to figure out how to convince doctors to live in smaller towns and cities and in less desirable provinces/states within a country. The source of the problem is that doctors are people and people have their own preferences and motivations for doing the things they do. Trying to convince even a handful of above average competency doctors to live in a remote region with bad weather and few amenities for more than a few years will be extremely difficult even if you offer them large sums of money (which aren't generally available anyways) because they have to weigh their own health and happiness in the balance when deciding where to live and practice.
A state should work to ensure everyone lives in a desirable area, doctors & patients.
I don't see how this is possible with the use of the state or not. I mean, I can throw out some ridiculous sci-fi ideas like geo-engineering and megaprojects that give every region the perfect balance of temperate seasons, agricultural productivity and variety, access to functional and esthetically pleasing waterways and pastoral landscapes that would make them all equally attractive places for people to live but that's just the definition of a utopia that will never exist.
You're also just copping out by saying "the state should handle it", who and what do you think the state is? It may come as a surprise that it's just a bunch of people who are just as imperfect and limited in their abilities as the rest. They can't simply wave their hands and make everyone happy. It make about as much sense as saying "Microsoft should handle it" or "the Catholic Church should handle it".
There are lots of places that are "desirable" to a large-enough-to-be-relevant portion of the population, but not as large a portion as the portion of the population that wants to be doctors. And they may like living there for reasons that someone who is drawn to a career in medicine might be unlikely to share.
>Is it impossible for highly advanced societies like ours to pay more for people to get trained as doctors, nurses or whatever is missing? Or to convince them to choose a profession that deals with other humans instead of UBI?
This is the point where it becomes important to distinguish two senses of "advanced", i.e. advanced in technological sense on the one hand and advanced in social/societal and especially large-group-long-time-horizon coordination terms on the other. In the former we are quite advanced, in the latter quite primitive and regressing by the day, it feels like. (But sorry to end on a doomer note, take it with a grain of salt.)
I remember as a kid being ill, you just called the doctor and they came to your house the same day, and you could just go to the doctor's practice hours without an appointment and sit their waiting your turn.
Now, I have to book an appointment online and the first available slot is in 2 weeks if lucky.
What exactly changed?
> if there is a shortage of doctors, why are we trying to solve that by training AI models modelled on influencers that spits out (hopefully improving) advice at 10x the rate of a human doctor?
The crucial part is the training. AI may very well be the solution for underserved communities, but not if it is trained on internet rubbish. Train an AI on curated state-of-the-art, scientific data, imagine the expert systems of yore on overdrive, and you will see much better results, including knowing when to call in a human doctor.
Making new doctors takes decades, lots of policy changes with uncertain outcomes, and demographic shifts nobody controls.
Using ChatGPT takes... nothing, it's already here.
Not too surprising to see one and not the other.
Also, the kinds of changes that result in more doctors don't tend to get media coverage. That's that boring, keep the lights on, politics that modern rage-bait driven media abhors, so it may even still be true that the changes we need for more doctors are also already happening. We'll find out in another decade or two.
Who is “we” here? Your healthcare needs are fulfilled by corporations, they are by design looking for scalable solutions with less human involved as possible.
I prepared for a new patient appointment with a medical specialist last week by role playing the conversation with a chatbot. The bot turned out to be much more responsive, inquisitive and helpful. The doctor was passive, making no suggestions, just answering questions. I had to prompt him explicitly to get to therapy recommendations, unlike the AI. I was glad that I had learned enough from the bot to ask useful questions. It would have been redundant if the doctor was active and interested, but that can't be depended on. This is standard procedure for me now.
I and many of my friends have used ChatGPT extremely effectively to diagnose medical issues. In fact, I would say that ChatGPT is better than most doctors because most doctors don't actually listen to you. ChatGPT took the time to ask me questions and based on my answers, narrowed down a particularly scary diagnosis and gave excellent instructions on how to get to a local hospital in a foreign country, what to ask for, and that I didn't have to worry very much because it sounded very typical for what I had. The level of reassurance that I was doing everything right actually made me feel less scared, because it was a pretty serious problem. Everything it told me was 100% correct and it guided me perfectly.
I was taking one high blood pressure medication but then noticed my blood sugar jumped. I did some research with ChatGPT and it found a paper that did indicate that it could raise blood sugar levels and gave me a recommendation for an alternative I asked my doctor about it and she said I was wrong, but I gently pushed her to switch and gave the recommended medication. She obliged, which is why I have kept her for almost 30 years now, and lo and behold, my blood sugar did drop.
Most people have a hard time pushing back against doctors and doctors mostly work with blinders on and don't listen. ChatGPT gives you the ability to keep asking questions without thinking you are bothering them.
I think ChatGPT is a great advance in terms of medical help in my opinion and I recommend it to everyone. Yes, it might make mistakes and I caution everyone to be careful and don't trust it 100%, but I say that about human doctors as well.
I agree that absolute deference to doctors is a mistake and that individuals should be encouraged to advocate for themselves (and doctors should be receptive to it) but I'm not so convinced in this specific case. Why do high blood sugar levels matter? Are there side effects associated with the alternative treatment? Has ChatGPT actually helped you in a meaningful way, or has the doctor's eventual relenting made you feel like progress has been made, even if that change is not meaningful?
In this context, I think of ChatGPT as a many-headed Redditor (after all, reddit is what ChatGPT is trained on) and think about the information as if it was a well upvoted comment on Reddit. If you had come across a thread on Reddit with the same information, would you have made the same push for a change?
There are quite a few subreddits for specific medical conditions that provide really good advice, and there are others where the users are losing their minds egging each other on in weird and whacky beliefs. Doctors are far from perfect, doctors are often wrong, but ChatGPT's sycophancy and a desperate patient's willingness to treat cancer with fruit feel like a bad mix. How do we avoid being egged on by ChatGPT into forcing doctors to provide bad care? That's not a rhetorical question, curious about your thoughts as an advocate for ChatGPT.
> Why do high blood sugar levels matter?
I have type 2 diabetes.
> How do we avoid being egged on by ChatGPT into forcing doctors to provide bad care?
I don’t ask it leading questions. I ask “These are my symptoms, give me some guidance.” Instead of “these are my symptoms, I think I have cancer. Could I be right?” If I don’t ask leading questions it keeps the response more pure.
I know what you mean and I would certainly not want to blindly "trust" AI chatbots with any kind of medical plan. But they are very helpful at giving you some threads to pull on for researching. I do think they tend a little toward giving you potentially catastrophic, worst-case possibilities, but that's a known effect from when people were using Google and WebMD as well.
Yeah, and "I found one paper that says X" is very weak evidence even if you're correctly interpreting X and the specific context in which the paper says it.
> Why do high blood sugar levels matter?
Are you asking why a side effect that is actually an entire health problem on its own, is a problem? Especially when there is a replacement that doesn’t cause it?
Side effects do not exist in isolation. High blood sugar is not a problem if it is solving a much bigger health issue, or is a lesser side effect than something more serious. If medication A causes high blood sugar but medication B has a chance of causing blood clots, medication A is an obvious choice. If a patient gets it in their head that their high blood sugar is a problem to solve, ChatGPT is going to reinforce that, whereas a doctor will have a much better understanding of the tradeoffs for that patient. The doctor version of the x/y problem.
I have type 2 diabetes. Blood sugar levels are a concern for me. I switched medications to one that was equally benign and got my blood sugar levels decreased along with my blood pressure. I don’t know why you assume that the medication I switched to might have higher or worse side effects. That wasn’t a choice I had to make given the options I was presented with.
Look, anyone can argue hypotheticals. But if one reads the comment being discussed, it can be deduced that your proposed hypotheses are not applicable, and that the doctor actually acknowledged the side effect and changed medications leading to relief. Now, if the new medication has a more serious side effect, the doctor (or ChatGPT) should mention and/or monitor for it, but the parent has not stated that is the case (yet). As such, we do not need to invent any scenarios.
The comment being discussed advocates for people to use ChatGPT and push their doctor to follow its recommendations. Even if we assume the OP is an average representation of people in their life, that means half of the people they are recommending ChatGPT to for medical advice are not going to be interrogating the information it provides.
A lazy doctor combined with a patient that lacks a clear understanding of how ChatGPT works and how to use it effectively could have disastrous results. A lazy doctor following the established advice for a condition by prescribing a medication that causes high blood sugar is orders of magnitude less dangerous than a lazy doctor who gives in to a crackpot medical plan that the patient has come up with using ChatGPT without the rigour described by the comment we are discussing.
Spend any amount of time around people with chronic health conditions (online or offline) and you'll realise just how much damage could be done by encouraging them to use ChatGPT. Not because they are idiots but because they are desperate.
As a physician, I can give further insight. The blood pressure medication the commenter is referring to is almost certainly a beta blocker. The effect on blood sugar levels is generally modest [1]. (It is rare to advise someone with diabetes to stop taking beta blockers, as opposed to say emphysema, where it is common)
They can be used for isolated, treatment of high blood pressure, but they are also used for dual treatment of blood pressure and various heart issues (heart failure, stable angina, arrhythmias). If you have heart failure, beta blockers can reduce your relative annual mortality risk by about 25%.
I would not trust an LLM to weigh the pros and cons appropriately knowing their syncophantic tendencies. I suspect they are going to be biased toward agreeing with whatever concerns the user initially expresses to them.
[1]
No, it was hydrochlorothiazide.
I replaced it with Lisinopril with no side effects.
> most doctors don't actually listen to you.
> doctors mostly work with blinders on and don't listen
This has unfortunately been my experience as well. My childhood PCP was great but every interaction I've had with the healthcare system since has been some variation of this. Reading blood work incorrectly, ignoring explanations of symptoms, misremembering medications you've been taking, prescribing inappropriate medications, etc. The worst part is that there are a lot of people that reflexively dismiss you as a contrarian asshole or, even worse, a member of a reviled political group that you have nothing to do with just because you dare to point out that The Person With A Degree In Medicine makes glaring objective mistakes.
Doctors aren't immune to doing a bad job. I don't think it's a secret that the system overworks them and causes many of them to treat patients like JIRA tickets - I'd just like to know what it would take for people to realize that saying such doesn't make you a crackpot.
As an aside I use Claude primarily for research when investigating medical issues, not to diagnose. It is equally likely to hallucinate or mischaracterize in the medical domain as it is others.
ChatGPT helped me understand a problem with my stomach that multiple doctors and numerous tests have not been able to shed any effective light on. Essentially I plugged in all of the data I could from my own observations of my issue over a 35 year period. It settled on these 3 possibilities: "Functional Dyspepsia with slow gastric accommodation, Mild delayed gastric emptying (even subclinical), Visceral hypersensitivity (your nerves fire pain signals when stretched)" and suggested a number of strategies to help with this. I implemented many of them and my stomach pain has been non-existent for months now... longer than I have ever been pain free.
I feel like the difference is that doctors took what I told them and only partially listened. They never took it especially seriously and just went straight to standard tests and treatments (scopes, biopsies and various stomach acid impacting medications). ChatGPT took some of what I said and actually considered it, discounting some things and digging into others (I said that bitter beer helped... doctor laughed at that, ChatGPT said that the alcohol probably did not help but that the bittering agent might and it was correct). ChatGPT got me somewhere better than where I was previously... something no doctor was able to do.
ChatGPT for health questions is the best use case I have found (Claude wins for code). Having a scratch pad where I can ask about any symptom I might feel, using project memory to cross reference things and having someone actually listen is very helpful. I asked about Crohn's disease since my grandfather suffered from it and I got a few tests I could do, stats on likelihood based on genetics, diet ideas to try and questions to ask the doctor. Much better than the current doc experience which is get the quickest review of my bloods, told to exercise and eat healthy and a see you in six months.
I’ve heard many people say the same (specifically about ppl being better than doctors because they listen) and I find it odd and wonder if this is a specific country thing?
I’ve been lucky enough to not need much beyond relative minor medical help but in the places I’ve lived always found that when I do see a GP they’re generally helpful.
There’s also something here about medical stuff making people feel vulnerable as a default so feeling heard can overcompensate the relationship? Not sure I’m articulating this last point well but it comes up so frequent (it listened, guided me through it step by step etc.) that I wonder if that has an effect. Feeling more in control than a doctor who has other patients and time constraint just say it’s x or do this
In America a side effect of our lack of universal care is that every physician has to carry their own malpractice insurance, whereas in most countries you can just get retreated if the first time doesn't work. The Dr might still face consequences is their was actual malpractice but there isn't the shakels of having to do it by the book so strictly.
You may have done so inadvertently. With the transition to doc-in-the-box urgent care with NPs with inconsistent training and massive caseloads... your provider is often using ChatGPT.
I caught one when i ask ChatGPT something, and then went to urgent case. I told my story, they left, came back and essentially read back exactly what ChatGPT told me.
I don't know if I could trust AI for big things, but I had nagging wrist pain for like a year, any time I extended my wrist (like while doing a pushup). It wasn't excruciating but it certainly wasn't pleasant, and it stopped me from doing certain activities (like pushups)
I visited my GP, 2 wrist specialists, and physical therapist to help deal with it. I had multiple x rays and an MRI done. Steroid injection done. All without relief. My last wrist specialist even recommended I just learn to accept it and don't try to extend my wrist too much.
I decided to ask Gemini, and literally the first thing it suggested was maybe the way I was using the mouse was inflaming an extensor muscle, and it suggested changing my mouse and a stretch/massage.
And you know what, the next day I had no wrist pain for the first day in a year. And it's been that way for about 3 weeks now, so I'm pretty hopeful it isn't short term
I guess it’s not nothing that Gemini caught that, but that seems like a pretty obvious oversight from the healthcare practitioners - inquiring about RSI injuries should be one of the first things they ask about.
I pre-emptively switched to trackballs and to alternating left/right hands for mousing near the start of my professional career based on the reading I did after some mild wrist strain.
I think that 90% of what we get from doctors on musculoskeletal injuries that aren't visible on a simple X-ray is either oversight, or a bias towards doing nothing specifically because treatments have health, administrative, and financial costs and they might not help. There is no time authorized to do deep diagnostic work unless something is clearly killing you.
I’ve had good results with doctors for soft tissue injuries, but it doesn’t feel like something that GPs are generally equipped/motivated for. The good results I’ve had have been from doctors at high performance sports clinics, or with doctors I’ve been referred to by my (awesome) sports physiotherapist.
Yeah my dad would always go to sports medicine docs specifically because GPs are basically going to tell you to stop whatever hurts but sports medics will tell you how to do it without pain.
I had progressively worsening pelvic floor pain issues that AI helped me with and are now in remission/repair. My decade of interaction with multiple urologists and clinicians could be characterized as repeated and consistent "pretty obvious oversight from the healthcare practitioners".
Nothing personal, but without receipts I am going to take this story as made up propaganda. I simply can't believe that visiting 4 health professionals, giving context about using a computer all day, nobody diagnosed a very common issue nor suggested it as a possibility.
I simply can't imagine this happening based on my experience with healthcare in two (European) countries.
I was having wrist pain from using the mouse, and switched to a trackball, issue solved, if the wrist doesn't move/flex, it doesn't get strained, therefore no pain.
The only thing that moves is my thumb, and it's much better for flexing than the wrist, also it has a tiny load to manage vs the wrist.
I did the same, though to non-thumb trackball (CST2545), which I find also virtually eliminates wrist stress.
+100 to this from my personal experience
No, no, no. You can change your doctor, and get one that listens to you - you can't change the fact that ChatGPT has no skin in the game - no reputation, no hippocratic oath, no fiscal/legal responsibility. Some people have had miracles with Facebook groups, or WebMD, but that doesn't change where the role of a doctor is or mean that you should be using those things for medical advice as opposed to something that allows you to have an informed conversation with a doctor.
Neither do most doctors. No gp will get disbarred for giving the wrong diagnosis on a first consult.
They have 15 minutes and you have very finite money.
Medical agents should be a pre consult tool that the patient talks to in the lobby while waiting for the doctor so the doctor doesn't waste an hour to hear the most important data point and the patient doesn't sit for an hour in the lobby doing nothing.
Indeed you might have to for a variety of reasons that have nothing to do with your quality of care. I’ve had to do so a few times now.
Doctors have no skin in the game too. Our society is built on the illusion of 'skin in the game' of professionals like doctors and lawyers (and to a lesser extent, engineers), but it's still an illusion.
Engineers to a _lesser_ extent? Maybe software engineers, but for every other engineer, theres very tangible results and they absolutely get sued when things go wrong, and the governing bodies strip engineers of their licenses for malpractice all the time.
Source: I used to be a geotechnical engineer and left it because of the ridiculous personal risk you take on for the salary you get.
In countries with public healthcare + doctor shortages (e.g. Canada), good luck even getting a family doctor, let alone having a request to switch you family doctor "when you already have one!" get taken seriously.
Everyone I know just goes to walk-in clinics / urgent-care centres. And neither of those options give doctors any "skin in the game." Or any opportunities for follow-up. Or any ongoing context for evaluating treatment outcomes of chronic conditions, with metrics measured across yearly checkups. Or the "treatment workflow state" required to ever prescribe anything that's not a first-line treatment for a disease. Or, for that matter, the willingness to believe you when you say that your throat infection is not in fact viral, because you've had symptoms continuously for four months already, and this was just the first time you had enough time and energy to wake up at 6AM so you could wait out in front of the clinic at 7:30AM before the "first-come-first-served" clinic fills up its entire patient queue for the day.
The US has the same issue.
Because the republican party turned out to be a bunch of fascist fucks, there's no real critique of Obamacare. One of the big changes with the ACA is that it allowed medical networks to turn into regional cartels. Most regions have 2-3 medical networks, who are gobbled up all of the medical practices and closed many.
Most of the private general practices have been bought up, consolidated to giant practices, and doctors paid to quit and replaced by other providers at half the cost. Specialty practices are being swept up by PE.
> no reputation, no hippocratic oath, no fiscal/legal responsibility.
To say nothing of giving your personal health information over to a private company with no requirement to practice HIPAA, and just recently got subpoenaed for all chat records. Not to mention potential future government requests, NSA letters, during an administration that has a health secretary openly talking about rounding up mentally ill people and putting them in work camps.
Maybe LLMs have use here, but we absolutely should not be encouraging folks to plug information into public chatbots that they do not control and do not run locally.
It is a recipe for disaster.
> No, no, no. You can change your doctor
As an American on ACA this made me chuckle.
Did you read the article?
This was not what I was expecting. The doctors I know are mostly miserable; stuck between the independence but also the burden of running their own practice, or or else working for a giant health system and having no control over their own days. You can see how an LLM might be preferable, especially when managing a chronic, degenerative condition. I have a family member with stage 3 kidney disease who sees a nephrologist, and there's nothing you can actually do. No one in their right mind would recommend a kidney transplant, let alone dialysis for someone with moderately impaired kidneys. All you can do is treat the symptoms as they come up and monitor for significant drops in function.
"You are my best health adviser!” she praised it once.
It responded: “Hearing you say that really makes me so happy! Being ..."
You should try praising a Claude instance after a successful session. It is eerie.
One of my kids recently had a no-contact knee injury while playing basketball. He immediately started limping and crying and I had to carry him from the court to the car.
I did some searching with Grok and I found out:
- no contact injuries are troubling b/c it generally means they pulled something
- kids don't generally tear an ACL (or other ligament)
- it's actually way more common for the ligament to pull the anchor point off of the bigger bone b/c kid bones are soft
I asked it to differentially diagnose the issue with the details of: can't hold weight, little to no swelling and some pain.
It was adamant, ADAMANT, that this was a classic case of bone being pulled off by the ligament and that it would require surgery. It even pointed out the no swelling could be due to a very small tear etc. It gave me a 90% chance of surgery too.
I followed up by asking what test would definitely prove it one way or the other and it mentioned getting an X-Ray.
We go off to the urgent care, son is already kind of hobbling around. Doctor says he seems fine, I push for an X-Ray and turns out no issue: he probably just pulled something. He was fully healed in 2-3 days.
As someone who has done a lot of differential diagnosing/troubleshooting of big systems (FinTech SRE) I find it interesting that it was basically correct in what could have happened but couldn't go the "final mile" to establish it correctly. Once we start hooking up X-Rays to Claude/Grok 4.2 etc equivalent LLMs, will be even more interesting to see where this goes.
>I did some searching with Grok
Grok is...not most people's first choice.
At least both OpenAI and Deep Mind do medical fine tuning, and both are almost certainly paying doctors to do it.
+1. I wish Gemini 2.5/3 Pro's "personality" and long context handling wasn't so erratic, because the medical stuff in there is great. Whatever they did to produce the MedGemma models is clearly built on a strong baseline. I haven't had need to try using MedGemma on x-ray imagery, but I'd be curious to hear results — imagery diagnostics is part of what it's built for.
Opus 4.5 seems good too, though getting dumber. OpenAIs fine tuning is clearly built to toe the professional medical advice line, which can be good and bad.
I like this post about a chat bot being 100% completely, confidently, adamantly wrong that characterizes it as being “basically right” about something that was untrue and did not happen.
It is like getting phished and then pointing out that the scammer was basically right about being Coinbase support aside from the fact that they did not work there
If you get the "You're absolutely right!" response from an LLM that screwed up on a field you're familiar with and still let them play with your health, you're...courageous to say the least.
Much like talking to your doctor - you need to ask/prompt the right questions. I've seen chatgpt and gemini make one false assumption that was never mentioned and run with it and continue referencing it down the line as if it were fact... That can be extremely dangerous if you don't know enough to ask it to reframe or verify, or correct it's assumption.
If you are using it like a tool to review/analyze or simplify something - ie explain risk stratification for a particular cancer variant and what is taken into account, or ask it to provide probabilities and ranges for survival based on age/medical history, it's usually on the money.
Every other caveat mentioned here is valid, and it's valid for many domains not just medical.
I did get hemotologist/oncologist level advice out of chatgpt 4o based on labs, pcr tests and symptoms - and those turned out to be 100% true based on how things panned out in the months that followed and ultimately the treatment that was given. Doctors do not like to tell you the good and the bad candidly - it's always "we'll see what the next test says but things look positive" and "it could be as soon as 1 week or as long as several months depending on what we find" when they know full well you're in there for 2 months at minimum you're a miracle case. Only once cornered or prompted will they give you a larger view of the big picture. The same is true for most professional fields.
For major medical issues it may well be best practice to use the four eyes principle like we do for all safety related systems. Access is key and at this time getting a second pair of eyes in close timely proximity is a luxury few have and even fewer will have looking at the demographics in the developed world. Human doctors are failable as is AI. For the time being having a multitude of perspectives may well be the best in most cases.
The big bank phrase for "four eyes" was "maker/checker" and I always enjoyed that it rhymed.
You are 100% correct that, much like pilots overseeing autopilots, we should combine the best of both worlds in the medical field.
The problem is not reliance on AI but that the AI is not ready yet and using general-purpose models.
There isn't simply enough doctors to go around and the average one isn't as knowledgeable as you would want. Everything suggests that when it comes to diagnosis ML systems should be better in the long run on average.
Especially with a quickly aging population there is no alternative if we want people to have healthcare on a sensible level.
They really have arrived just in time, I think.
Considering how difficult it is to get patients to talk to doctors, using AI can be a great way to get some suggestions and insight _and then present that to your actual doctor_
Patients talking to doctors is the easy part. Doctors actually listening to anything they don't already think is nearly impossible.
It really hit me when she said the chatbot was more human, the doctor was like a machine. I heard the same complaint from my mother about her doctor, and about being afraid to ask a question because he might snap at her.
Not a single response how the doctors got her all this data to be able to be fed into DeepSeek?
AI is doing the last mile and all these posters are saying it’s replacing doctors. Scary stuff.
The dangers are obvious (and also there are some fascinating insights into how healthcare works practically in China). I wonder if some kind of "second opinion" antagonistic approach might reduce the risks.
Medical Advice Generative Adversarial Networks would be a good idea.
I see some of this adversarial second-guessing introspection from Claude sometimes. ("But wait. I just said x y and z, but that's inconsistent with this other thing. Let me rethink that.")
Sometimes when I get the sense that an LLM is too sycophantic, I'll instruct it to steelman the counter-argument, then assess the persuasiveness of that counter-argument. It helps.
Can you expand on the insights into the chinese medical system ?
I hadn't really even seen much in the way of recent, first / second person accounts of what it's like to receive chronic care in modern China. It's sad but maybe not surprising to see that it's dysfunctional in its own special way.
The basic calculus is that if you decrease the number of minutes of doctor time enough, it doesn't really matter if they are 100% correct (0 minutes of 100% correct = 0 advice).
Patients are substituting 15 minutes of 100% correct information for 2 hours of 80%, or whatever percentage the AIs are.
Additionally, they're pretty much free at the moment and available 24/7 with no travel required.
The problem with any patient to doctor or patient to Dr. Claude is that intentionally or not the patient will lie or misinterpret their condition, to the point of not describing it properly, risking getting a bad diagnosis or suggestion with a chatbot. In a classical medical scenario this can still happen but there is the chance that the medical professional could see through to the actual problem. There is a reason why even in real life it's sometimes best to get a second medical opinion.
Chatbots aren't better than doctors can be. But in the US, doctors are a highly credentialed position, therefore expensive, therefore their time is split into minimal parcels. (Still longer than 5 minutes at least!!) In my experience, chatbots are often better than doctors are in the real world, at least for savvy users.
Reminds me of an excellent paper I just read by a former Google DeepMind Ethics Research Team member
https://www.mdpi.com/2504-3900/114/1/4 - Reinecke, Madeline G., et al. "The double-edged sword of anthropomorphism in llms." Proceedings. Vol. 114. No. 1. MDPI, 2025 Author: https://www.mgreinecke.com/
Worriesome for sure.
However I would say that the cited studies are somewhat outdated already compared e.g. with GPT-5-Thinking doing 2mins of reasoning/search about a medical question. As far as I know Deepseeks search capabilities are not comparable and non of the models in the study spend a comparable amount of compute answering your specific question.
What a nonsense, sorry...
Human is human already, regardless how they speak and what they do... There's always true sympathy and empathy to find in a human, regardless how busy, dark, erroneous etc. they are... Yet, it's the true alive soul inside in every person...
Medics are likely tired, doing their job every single day, yet they have empathy, and will always try to listen if you actually try... They know what PAIN, AGONY, DEATH, SORROW... fear means...
LLM/"AI" will always pretend to be a human, since it's "trained"/designed, to be so, and will always be limited and incomplete... Not to mention the initial dataset of numerous emphatic actual human has its limited memory inside, no one is responsible for.
What a hopeless sorrow is that awful trendy, advertised, mind-atrophying mess...
"Doctors are more like machines"
"Machines are more like humans"
I love the future...
A sick family member told me something along the lines of, "I know how to work with AI to get the answer." I interpret that to mean he asks it questions until it tells him what he wants to hear.
Indeed, real doctors have the advantage of understanding how to treat humans that are incapacitated. =3
https://www.youtube.com/watch?v=yftBiNu0ZNU
Just seeing that guy’s face and hearing his voice makes me uneasy. That channel is total body-horror. Glad that guy ended up okay, unlike this poor soul:
https://youtube.com/watch?v=NJ7M01jV058
Indeed, the instinctual pucker factor for some things in this world are warranted. That case is famous, and still controversial to those with inflated hubris. =3
This almost certainly isn't only a China problem. I've observed UK users asking questions about diabetes and other health advice. We also have an inexpensive (free-at-point of use for most stuff) but stretched healthcare system. Doubtless there are US users looking at the cost of their healthcare and resorting to ChatGPT instead too.
In companies people talk about Shadow-IT happening when IT doesn't cover the user needs. We should probably label this stuff Shadow-Health.
To some extent, the deployment of a publicly funded AI health chat bot, where the responses can be analysed by healthcare professionals to at least prevent future harm is probably significantly less bad than telling people not to ask AI questions and consult the existing stretched infrastructure. Because people will ask the questions regardless.
The joke of looking symptoms up on WebMD and determining you have cancer has been around for... geez over 20 years now. Anti-vaccine sentiment mostly derived from Facebook. Google any symptom today and there are about 10 million Quora-esque websites of "doctors" answering questions. I'm not sure that funneling all of this into the singular UI of an AI interface is really better or worse or even all that different.
But I do agree that some focused and well funded public health bot would be ideal, although we'll need the WHO to do it, it's certainly not coming from the US any time soon.
It's funny how he cites all these unreliable sources his mother used to consult but who's to say DeepSeek wasn't trained on them?
It's still insane to me how people can trust an LLM they didn't train themselves! You have no idea what vile and evil Dreck these models were trained with.
It's the click workers in third world countries who you don't recognize exist who could know if they were experts. But they aren't. A huge responsibility for these underpaid workers. They are the backbone of this hype, not GPUs. It's all built on shit. :)
Relevant:
The AI Revolution in Medicine: GPT-4 and Beyond by Peter Lee (Microsoft Research) - https://www.microsoft.com/en-us/research/publication/the-ai-...
The AI Revolution in Medicine, Revisited by Peter Lee (Microsoft Research) - https://www.microsoft.com/en-us/research/story/the-ai-revolu...
> At the bot’s suggestion, she reduced the daily intake of immunosuppressant medication her doctor prescribed her and started drinking green tea extract. She was enthusiastic about the chatbot
I don't know enough about medicine to say whether or not this is correct, but it sounds suspect. I wouldn't be surprised if chatbots, in an effort to make people happy, start recommending more and more nonsense natural remedies as time goes on. AI is great for injuries and illnesses, but I wonder if this is just the answer she wants, and not the best answer.
As soon as the model detects user pleasure at not needing a scary surgery (especially if you already confided that you're scared) then it'll double down on that line to please the user.
med student here: Reducing immunosuppressant not something to be taken lightly for kidney transplant patients. I was shocked when I read that sentence.
I feel like we're in the part of the dystopian SciFi movie where they explain how civilization discovered some technological advance that they thought would be a panacea to all their woes. And despite not really understanding it or it's limitations, just started slapping it on absolutely everything, and before they know what happened everything comes crashing down.
We are witnessing the Great Filter in action.
(and not just with the AI stuff)
Access trumps everything else. A doctor is fine with you dying while you wait on his backlog. The machine will give you some wrong answers. The mother in the story seems to be balancing the concerns. She has become the agent of her own life empowered by a supernatural machine.
> She understood that chatbots were trained on data from across the internet, she told me, and did not represent an absolute truth or superhuman authority. She had stopped eating the lotus seed starch it had recommended.
The “there’s wrong stuff there” fear has existed for the Internet, Google, StackOverflow. Each time people adapted. They will adapt again. Human beings have remarkable ability to use tools.
I think the article can basically be summed up as "GenAI sychophancy should have a health warning similar to social media". It's a helluva drug to be constantly rewarded and flattered by an algorithm.
I think that's doing the article a disservice. "Some patients prefer the service of AI doctors to real doctors, even though they definitely do get a lot wrong" is what I got out of it.
Personally I think the article spends a lot of time trying to show that AI may be able to improve health outcomes, particulaly for rural patients, but IMHO it doesn't spend nearly enough time talking about the current challenges