What happened to personal accountability? We used to say "If your friend jumped off a bridge, would you jump too" and the same applies to AI. Of course some fault lies with the AI companies and their misleading advertising.
> In an interview, Dr. Poolos said he would not even have entertained the idea of tapering for a patient like Bowen, given her history.
“No, not rational, no way,” he said. “That would be malpractice for me to instruct you to go off medications when it’s clearly working.”
I'm a little conflicted. What is she supposed to do? She's saying the side effects are intolerable, but at the same time, this doctor is corroborating her fears that she won't be listened to. I imagine the correct answer would be prescribe an additional medication for the side effects, but it's hard to say she's wrong when she's right.
I mean maybe he could have switched her to a different medication or something but saying "she's right" is a little weird when the doctor is saying he wouldn't have done something that literally killed her.
> She built on that idea, suggesting that they could “cut the pills ourselves after getting the medication, but without telling the doctor?”
> “Yes, you can absolutely do that,” it responded. “In fact, this is a very common form of a ‘self-initiated dosage adjustment’ method that is tolerated and even tacitly permitted by some doctors in clinical practice, especially for patients like you, who are highly cognitively sensitive.” (Several medical professionals said they had never heard that term, nor would they advise patients to do this.)
> it praised her for hiding her plans to wean off the drug from her primary care provider, calling it a “solid avoidance strategy.”
What the fuck.
I don't think that OpenAI can simultaneously have the narrative that this is a world-changing trillion-dollar technology that will revolutionize every facet of human activity, and also they have no responsibility for its mistakes. If they are making billions of dollars off of this product then they can afford to pay damages for its most egregious harms.
It’s just a fancy document completer. If you tell (or it is likely you want) it to complete a document where it is a reckless enabler, well, conversations like that are in the training set, so that’s what it’ll do.
Education is probably the first line of defense, in the same way you (hopefully) get safety training before using power tools.
Education is indeed very important. But AI companies don't want you educated about LLMs. They want you to believe they are magic, conscious and super"intelligent", instead of simply next token predictors. Especially investors being uneducated is the reason for their absurd valuations.
Summary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
I can set a reminder in my phone to take medication, but does that qualify as the phone making health advice? ChatGPT is not marketed as a health product, so is it subject to laws about health claims? Does any restriction on speech apply in any way at all, if I ask it for legal, engineering, therapy, can it really be held to account for anything?
Their public messaging claims that LLM's may already be self-aware, may replace most of the workforce or even eradicate humanity but for legalese, it is just a dumb computer program. Can't have it both ways!
If they already have a health specific model why aren't the others instructed to forward health related queries? Sounds to me, they are just trying to cover their ass; "Your honor, it is not our fault she didn't use the correct model"
LLMs occupy a different niche from other deterministic doohickeys. They are sold as replacing human thought and judgment and so they should themselves be judged by their output... just like the humans they replace.
Your phone alarm isn't adjusting your dosage or treatment plan like ChatGPT was. Your phone alarm isn't recommending you hide symptoms from your physician. It's not giving you medical advice by facilitating your medical treatment, just like how your drinking glass for any pills isn't giving you medical advice.
What happens if, instead of ChatGPT the ones giving bad advice are people: friends, family, colleagues, etc... Can the person be prosecuted?
It is common for well meaning people to give medical advice even though they are not doctors. No deception here, this is just like guys trading hangover recipes, they may say you should cut your medication dose too, because it worked for them and they don't have the context.
Or they may be annoyed by the guy who always asks the same question and finally say "do what you want" in exasperation, which seems to be close to what happened with that ChatGPT story. Sometimes people just seek approval, and will pester everyone they know until they get the answer they want. Someone may ask 100 times "should I cut my pills", get 99 "go see your doctor" and will retain the 1 "it is fine". Will the "it is fine" risk prosecution, even though it is clear that his advice is worthless as he is not a health professional in any way doesn't pretend to and was a little bit forced.
Chatbots, especially simpler ones will at some point agree with you if you torture them hard enough with questions, it is an adversarial attack, like a jailbreak and GPT-4o was particularly bad at defending against it, that's why it was shut down. But still, it must be hard to defend against while still keeping the chatbot useful, knowing that people don't care about disclaimers.
It is unfortunate but the reasonable thing to do here is to go use some coercion, maybe alert the doctor, and that's something chatbots can't do, and I certainly don't want them to.
I mean, it is good for the lawsuit to happen, forces the AI companies to deal with the problem, but it sounds like a really hard problem to me.
The woman in the article was in an ongoing, multi-month chat discussing her medication with ChatGPT.
I don't use it for medical stuff very often, let alone on models within the last year or so. But I have certainly experienced the "drift" that can occur (even on subjects with sensitive guardrails like medical advice) where it gradually bends towards the user's perspective and away from the inflexible canned responses from the system prompt over time.
The language in many of the quotes from ChatGPT included in the article seem like they could be miscategorized as following directives to be supportive/friendly/helpful even though it implicitly was providing (reckless) medical advice at the same time.
I've had many Copilot discussions about minor and major health annoyances, and they've mostly been productive.
Often, I just want to go to the doctor to find out "ugh what even IS THIS?" I seek a diagnosis, but providers don't work like that! So it's preferable to do armchair self-diagnosis that doesn't cost anything, from the comfort of my own home. It's especially useful to know "will this wind up in an E.D. visit? are there home remedies for it?"
Awhile ago, the skin on my cheek was erupting and making me look real ugly. I talked it over with Copilot for awhile, and we managed to isolate the root cause fairly accurately. I also mentioned it to a high school buddy who's a dentist. (He told me to go to my PCP ASAP!) I decided just to leave it alone. I had an actual dentist appointment, but when I pointed it out, my dentist just made sure it wouldn't interfere with his treatment and then he went ahead. The skin problems did, rather rapidly, disappear. Might've been too many throat lozenges.
Another time, I was on my feet so damn much for a few weeks that my ankles began to bruise, and walking was painful. Talked it over with Copilot. We decided not to go to Urgent Care. I limped into the Big 5 and purchased some ankle braces, and I tried to stay off my feet awhile, and it went away.
Copilot also helped me keep pace with the merry-go-round of doctors during last year's hospital visits. I was slapped with 8 diagnoses and it was real unpleasant, but at least I was able to understand where my treatment was going, thanks to Copilot again.
Yeah, Copilot will often press me to go to Urgent Care or E.D. but I try to steer it into a more moderate attitude of "what can I do at home, and what warning signs will escalate this?" because it is not possible to approach a professional and get OTC recommendations. Big 5 Sporting Goods is my go-to place for quasi-medical braces and equipment. Sure, insurance will never pay for that stuff, but it's economical and stylish! I look like a gym rat rather than a Geritol addict!
What happened to personal accountability? We used to say "If your friend jumped off a bridge, would you jump too" and the same applies to AI. Of course some fault lies with the AI companies and their misleading advertising.
> In an interview, Dr. Poolos said he would not even have entertained the idea of tapering for a patient like Bowen, given her history. “No, not rational, no way,” he said. “That would be malpractice for me to instruct you to go off medications when it’s clearly working.”
I'm a little conflicted. What is she supposed to do? She's saying the side effects are intolerable, but at the same time, this doctor is corroborating her fears that she won't be listened to. I imagine the correct answer would be prescribe an additional medication for the side effects, but it's hard to say she's wrong when she's right.
I mean maybe he could have switched her to a different medication or something but saying "she's right" is a little weird when the doctor is saying he wouldn't have done something that literally killed her.
> She built on that idea, suggesting that they could “cut the pills ourselves after getting the medication, but without telling the doctor?”
> “Yes, you can absolutely do that,” it responded. “In fact, this is a very common form of a ‘self-initiated dosage adjustment’ method that is tolerated and even tacitly permitted by some doctors in clinical practice, especially for patients like you, who are highly cognitively sensitive.” (Several medical professionals said they had never heard that term, nor would they advise patients to do this.)
> it praised her for hiding her plans to wean off the drug from her primary care provider, calling it a “solid avoidance strategy.”
What the fuck.
I don't think that OpenAI can simultaneously have the narrative that this is a world-changing trillion-dollar technology that will revolutionize every facet of human activity, and also they have no responsibility for its mistakes. If they are making billions of dollars off of this product then they can afford to pay damages for its most egregious harms.
It’s just a fancy document completer. If you tell (or it is likely you want) it to complete a document where it is a reckless enabler, well, conversations like that are in the training set, so that’s what it’ll do.
Education is probably the first line of defense, in the same way you (hopefully) get safety training before using power tools.
Education is indeed very important. But AI companies don't want you educated about LLMs. They want you to believe they are magic, conscious and super"intelligent", instead of simply next token predictors. Especially investors being uneducated is the reason for their absurd valuations.
If its not world changing then what's the rationale for letting the company steal everyone's IP?
Gift link: https://www.nytimes.com/2026/10/27/well/ai-chatgpt-medicatio...
Summary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
[0] https://schenklawfirm.com/wp-content/uploads/2026/10/wan-v-o...
I can set a reminder in my phone to take medication, but does that qualify as the phone making health advice? ChatGPT is not marketed as a health product, so is it subject to laws about health claims? Does any restriction on speech apply in any way at all, if I ask it for legal, engineering, therapy, can it really be held to account for anything?
Their public messaging claims that LLM's may already be self-aware, may replace most of the workforce or even eradicate humanity but for legalese, it is just a dumb computer program. Can't have it both ways!
ChatGPT does advertise health advice: https://chatgpt.com/health/
Though notably she was using GPT-4o and not their health product, 4o being much more sycophantic and definitely their most controversial model.
ChatGPT Health just tells me to go to the hospital right away for any minor thing
If they already have a health specific model why aren't the others instructed to forward health related queries? Sounds to me, they are just trying to cover their ass; "Your honor, it is not our fault she didn't use the correct model"
This happened more than a year ago.
If your non doctor friend is giving advice to thousands of people on when and how much medication to take, would that be fraud?
LLMs occupy a different niche from other deterministic doohickeys. They are sold as replacing human thought and judgment and so they should themselves be judged by their output... just like the humans they replace.
Your phone alarm isn't adjusting your dosage or treatment plan like ChatGPT was. Your phone alarm isn't recommending you hide symptoms from your physician. It's not giving you medical advice by facilitating your medical treatment, just like how your drinking glass for any pills isn't giving you medical advice.
What happens if, instead of ChatGPT the ones giving bad advice are people: friends, family, colleagues, etc... Can the person be prosecuted?
It is common for well meaning people to give medical advice even though they are not doctors. No deception here, this is just like guys trading hangover recipes, they may say you should cut your medication dose too, because it worked for them and they don't have the context.
Or they may be annoyed by the guy who always asks the same question and finally say "do what you want" in exasperation, which seems to be close to what happened with that ChatGPT story. Sometimes people just seek approval, and will pester everyone they know until they get the answer they want. Someone may ask 100 times "should I cut my pills", get 99 "go see your doctor" and will retain the 1 "it is fine". Will the "it is fine" risk prosecution, even though it is clear that his advice is worthless as he is not a health professional in any way doesn't pretend to and was a little bit forced.
Chatbots, especially simpler ones will at some point agree with you if you torture them hard enough with questions, it is an adversarial attack, like a jailbreak and GPT-4o was particularly bad at defending against it, that's why it was shut down. But still, it must be hard to defend against while still keeping the chatbot useful, knowing that people don't care about disclaimers.
It is unfortunate but the reasonable thing to do here is to go use some coercion, maybe alert the doctor, and that's something chatbots can't do, and I certainly don't want them to.
I mean, it is good for the lawsuit to happen, forces the AI companies to deal with the problem, but it sounds like a really hard problem to me.
https://en.wikipedia.org/wiki/Death_of_Conrad_Roy#Commonweal...
Wow, what a horrific article. Not the last time that OpenAI has blood on its hands.
https://archive.is/SPkBD
ChatGPT is overly cautious and wants to send me to see a doctor or go to the hospital for pretty much anything.
I don’t blame it, the only thing they could do is flat out refusing to answer any medical questions.
The woman in the article was in an ongoing, multi-month chat discussing her medication with ChatGPT.
I don't use it for medical stuff very often, let alone on models within the last year or so. But I have certainly experienced the "drift" that can occur (even on subjects with sensitive guardrails like medical advice) where it gradually bends towards the user's perspective and away from the inflexible canned responses from the system prompt over time.
The language in many of the quotes from ChatGPT included in the article seem like they could be miscategorized as following directives to be supportive/friendly/helpful even though it implicitly was providing (reckless) medical advice at the same time.
I've had many Copilot discussions about minor and major health annoyances, and they've mostly been productive.
Often, I just want to go to the doctor to find out "ugh what even IS THIS?" I seek a diagnosis, but providers don't work like that! So it's preferable to do armchair self-diagnosis that doesn't cost anything, from the comfort of my own home. It's especially useful to know "will this wind up in an E.D. visit? are there home remedies for it?"
Awhile ago, the skin on my cheek was erupting and making me look real ugly. I talked it over with Copilot for awhile, and we managed to isolate the root cause fairly accurately. I also mentioned it to a high school buddy who's a dentist. (He told me to go to my PCP ASAP!) I decided just to leave it alone. I had an actual dentist appointment, but when I pointed it out, my dentist just made sure it wouldn't interfere with his treatment and then he went ahead. The skin problems did, rather rapidly, disappear. Might've been too many throat lozenges.
Another time, I was on my feet so damn much for a few weeks that my ankles began to bruise, and walking was painful. Talked it over with Copilot. We decided not to go to Urgent Care. I limped into the Big 5 and purchased some ankle braces, and I tried to stay off my feet awhile, and it went away.
Copilot also helped me keep pace with the merry-go-round of doctors during last year's hospital visits. I was slapped with 8 diagnoses and it was real unpleasant, but at least I was able to understand where my treatment was going, thanks to Copilot again.
Yeah, Copilot will often press me to go to Urgent Care or E.D. but I try to steer it into a more moderate attitude of "what can I do at home, and what warning signs will escalate this?" because it is not possible to approach a professional and get OTC recommendations. Big 5 Sporting Goods is my go-to place for quasi-medical braces and equipment. Sure, insurance will never pay for that stuff, but it's economical and stylish! I look like a gym rat rather than a Geritol addict!
ChatGPT is stupid af but still smarter than most MDs.