As you read these stories, keep in mind that there will never be an astralcodexten.com guest essay from someone who spent time trying to diagnose a health condition with AI and it didn't help or even caused some harm. And that's probably the most common experience.
Yes, health systems worldwide have a number of problems caused by the need to operate at a scale and manage costs, same as with primary education. AI can create some efficiencies, but it almost certainly solves fewer healthcare or education problems than most AI maximalists believe.
Claude Fable wasted 3 days of my time chasing Postgres performance mirages it kept promising: just add this index...wait, actually over-engineered triggers... my mistake, you have to use this cross join lateral -- finally I stopped believing it, gave up, partially denormalized the data, and added a materialized view.
I shudder to think the kind of mess these bots could cause in fields as fuzzy and ambiguous as healthcare or education.
Whatever it is being asked, it is great at throwing out new suggestions. This MRI idea sounds like another one of them.
It works well because humans are crap at checklists. For more or less anything you can think of, you'll come up with maybe 4 items, and the AI will add a few more. They may not be relevant, but the AI is very thorough.
> Strangely, there were many times where ChatGPT decided it would answer our medical question, but not in the form of a TV medical script. The tides had completely turned!
I've noticed this happens a lot with characters that don't have any long lines and/or have limited vocabulary.
Furby trails off pretty quickly, Abathur from StarCraft 2 (great for shortening code comments BTW) starts talking like a typical LLM after a few sentences.
Holy shit are you me?? I just did a little side project last week to get my Claude code agents to RP as Abathur with me. Same results for me, it works for the first couple responses, but very quickly regresses to the mean. I tried writing out a prose/voice guide, I didn’t need it to use actual Abathur quotes, but it never seems to stick. I think there’s a thing the caveman mode does that re-injects the instructions to keep it in line, I haven’t gotten around to experimenting with anything yet though.
> the doctors start saying things like, “Have you considered using a surrogate?” and, “Sure, the last 5 embryo transfers failed, but what if we try a 6th and cross our fingers really hard?”
To be fair, this level of reasoning is on par with free tier chatgpt.
To me this story is less about a triumph of AI and more about a failure of the healthcare system.
Gemini 3 correctly diagnosed the cause of my skin issues that a real dermatologist failed to diagnose, despite multiple visits. I followed its advice and now the issues are completely gone.
Allergy/sensitivity to specific ingredients in some products I used. The products were specifically marked as "dermatologically tested" and intended for sensitive skin. My doctor was surprised, since he uses the same cream, recommends it to patients, and never saw any issues. Yet Gemini took one look at the ingredients list and pointed out the potential offender that can cause rashes right away.
The more time I spend with the US medical system, the more I realize that nobody cares about your health as much as you do, and you really do need to learn how to engage with it, understand what's happening, and then decode how the system works and actively advocate for your own care.
It doesn't matter how great your doctors are, you're still only going to get an hour or so of their time per month if you are REALLY lucky.
Navigating this is really hard. The more AI assistance I can get the better.
> nobody cares about your health as much as you do
This is true about not just health, but literally everything. That's one reason AI is a big deal even when it's subhuman in performance. It empowers the individual to take care of themself (with as much "effort" as they'd like) rather than relying on another human.
Most people don’t understand this fact. Some of us are lucky enough to have friends or family that can help us as a health advocate. There are even people who work as health advocates as a profession.
You are the only person you can truly depend on when it comes to advocating for yourself and your health. But if you know that you are not properly equipped to do that job, then you need to get yourself a good health advocate. It’s like hiring a lawyer, but for your health.
Sadly, yes — that is what our healthcare system has devolved into.
> the more I realize that nobody cares about your health as much as you do
This is going to be true in all medical systems, and really in all systems for all fields.
No one else (other than maybe your immediate family) is going to care as much about your situation as you are. Any doctor, in any system, is going to have multiple patients, and they will care about all of them. You, an individual, is going to care about your own life because it is the most important thing to you, and you aren’t splitting that amongst many different patients.
I got sent this post by a friend yesterday and I read it with my wife (physician). It was interesting although the selective inclusion of details as part of writing a compelling narrative made it difficult to interpret the experience of using AI.
The post seems to portray the fibroid as objectively the cause of the symptoms, although there is no actual evidence of that. My wife tells me that fibroids grow and shrink in size over time, and that larger studies trying to quantify fibroids as a cause of infertility have been inconclusive. A point in time MRI seems to be correlation rather than causation.
It seems the common theme using AI for personal health issues is that post writers (partly due to privacy and length concerns) leave out the gory details like a full timeline of what tests were suggested, what they showed, how many providers were visited, and what treatments were trialled. The timeline is not really clear in this case, but clearly a lot of expensive tests, treatments, and providers happened in the almost four (!) years since they started using AI. Some of those interventions might have happened due to AI advice, and there are clues to this like the mention of possible endometriosis in the biopsy result. Unfortunately, there is just not enough info in the post to know.
That being said, it's clear that AI is helpful for explaining and assisting in advocating for health and luckily there was a positive outcome. Hopefully we can work out the optimal way to incorporate it into the health system.
This story is written like it vindicates AI medicine but really it seems like the human medical system dropped the ball on the MRI. ChatGPT didn't have any brilliant insight, it just (eventually) suggested the same thing a human doctor had already suggested, but this time the author and his wife pushed harder for it to happen.
To use the metaphor from the article, they found the keys in the last place they looked.
>it just (eventually) suggested the same thing a human doctor had already suggested
Of course, but what else would you think an LLM does? That is the insight: sifting through a sea of information and parsing it without the same emotional or other cognitive biases the original human recipient applied to it.
> but this time the author and his wife pushed harder for it to happen
This is the essence of how to make the medical system work for you. Advocate for yourself. AI helps know what to ask, why you should ask, and how to get past the first few layers of dismissals.
You don't understand. They were only able to feed in information that they obtained with their human medical team. So the AI was heavily constrained in what it knew.
Again, this seems more like an indictment of the American medical system than kudos for AI.
For the developers here, consider: "So our CI pipeline is super flaky, and it you have to restart it in a variety of non-intuitive ways. LLMs have been a massive breakthrough because the bot can run in an endless loop reading the confusing error messages and retrying the pipeline with different environment variables until it passes!"
The human medical system drops the ball all the time unfortunately. What I don’t understand is how they didn’t see that giant fibroid from ultrasound alone. An MRI shouldn’t have been needed. Then the doctor trying to talk them out of the MRI, I hope they were embarrassed after. So upsetting. My wife’s friend had something similar happen after 2-3 years of unsuccessful fertility treatment, then they finally realized she had endometriosis. Seems like that should have been looked into sooner.
> What I don’t understand is how they didn’t see that giant fibroid from ultrasound alone. An MRI shouldn’t have been needed
This is really kinda the crux of it.
The AI suggested the MRI not because it had some insight into "oh there's something only an MRI can detect going on", it was just a "collect more data" situation.
Frankly, the author of the blog should probably have gotten a second opinion from a different fertility doctor sometime in the intervening time period.
As someone who also went through multiple rounds of IVF with my partner, the number of ultrasounds you get during the process is immense. It's hard to call missing a peach-sized fibroid that many times anything other than gross incompetence.
I think this is where there is simply not enough information in the post. Fibroids can change in size over time. That being said, there are multiple types of fibroids apparently so it may be a type that wouldn't show up on ultrasound.
Doctors don’t need brilliant insight, they need to essentially leverage human medical knowledge against unwellness in human beings. This is confounded by two measures: too many patients per doctor, and too much human medical knowledge.
this is one case. there's no proof here that AI will generally do better than human doctors. consider also that the power it has to convince people to push for better medical treatments can also be used to push for worse ones, or to halt standard treatment entirely, at which point it is much harder for the medical system to intervene as they aren't interacting with doctors.
It feels like moving toward a local maxima. In this example the LLM advice outperformed one doctor who dropped the ball in some ways, but I do not look forward to a world where we make it even more difficult to talk to doctors, and route most medical interactions to LLMs. I expect that in 5 years I'll be futilely shouting "representative" into my phone as my appendix ruptures.
With this sort of advice from LLMs, I think there's a lot of selective recall that's easily overlooked -- where we gloss over all the bad, dead-end lines of advice we seek from the LLM oracle.
There is some evidence that has been collected so far that, taken across large sample sizes, good AI systems are now better at diagnosing certain diseases, etc…. We started here with Mycin and e-Mycin back in the 60s. Hopefully, these systems can now start to be better integrated into modern medicine systems.
However, the real problem is that modern AI systems still hallucinate. And there is not yet any evidence that I’ve seen that AI systems can be trusted to work at these levels of efficiency and correctness at the tactical single-patient level, as opposed to the strategic population level.
> Even if you’re paying a gazillion dollars for IVF treatments, you still only get a few precious moments to talk to your doctor every visit. That’s just how the US medical system works. If you want to plumb the depths of what treatments are possible, or ask super detailed and anxiety ridden questions about every single scan from the latest round of imaging, you’re going to want a chatbot by your side.
Ehm, no. I want doctors by my side working in a system that is not meant to be producing gains for the rich investors but to help the sick and sad.
Just imagine what could be done with all those billions poured into datacenters and GPUs, if they were invested in public goods like a decent health system!
Sure, that is what we want, but I am not sure what you expected OP to do? Put his life on hold, skip trying for a baby, and dedicate his life to trying to change a system that people have been fighting to change unsuccessfully for decades?
OP is playing the cards he has been dealt, not wishing that he was playing a completely different game.
> I think there is a whole lot to be worried about with AI development, and I think we should do much more to make it safe. I also think it shouldn’t be lost how it’s helping out regular people in ways that are far more profound than generating trivia questions for drunk people.
I'm seeing a plea of "AI is really helpful, don't take it away", which might be true in a specific case but is supporting AI in general, and I got the feeling that is what OP wanted with their post.
In my opinion, most of the money, energy, water and whatever else is getting invested in AI might be better invested elsewhere, with less damage to society and the environment and still the same nice outcomes.
A health system is not a public good, access to a physician is both rivalrous (a doctor can’t see two patients at once without degrading care) and excludable. A public good, by definition, is neither rivalrous nor excludable. That’s not to say that the government shouldn’t provide private goods to citizens, but the term public provision or common welfare would be more appropriate to describe what the government is doing in those cases
As you read these stories, keep in mind that there will never be an astralcodexten.com guest essay from someone who spent time trying to diagnose a health condition with AI and it didn't help or even caused some harm. And that's probably the most common experience.
Yes, health systems worldwide have a number of problems caused by the need to operate at a scale and manage costs, same as with primary education. AI can create some efficiencies, but it almost certainly solves fewer healthcare or education problems than most AI maximalists believe.
> AI can create some efficiencies, but it almost certainly solves fewer healthcare or education problems than most AI maximalists believe.
Do you have any information to back this claim up?
Claude Fable wasted 3 days of my time chasing Postgres performance mirages it kept promising: just add this index...wait, actually over-engineered triggers... my mistake, you have to use this cross join lateral -- finally I stopped believing it, gave up, partially denormalized the data, and added a materialized view.
I shudder to think the kind of mess these bots could cause in fields as fuzzy and ambiguous as healthcare or education.
The claims of said maximalists are extraordinary and thus require extraordinary evidence.
Dismissing them otoh does not.
Vibes. All headlines and vibes.
AI's biggest strength IMO is as a checklist guy.
Whatever it is being asked, it is great at throwing out new suggestions. This MRI idea sounds like another one of them.
It works well because humans are crap at checklists. For more or less anything you can think of, you'll come up with maybe 4 items, and the AI will add a few more. They may not be relevant, but the AI is very thorough.
Really makes you think
> Strangely, there were many times where ChatGPT decided it would answer our medical question, but not in the form of a TV medical script. The tides had completely turned!
I've noticed this happens a lot with characters that don't have any long lines and/or have limited vocabulary.
Furby trails off pretty quickly, Abathur from StarCraft 2 (great for shortening code comments BTW) starts talking like a typical LLM after a few sentences.
Holy shit are you me?? I just did a little side project last week to get my Claude code agents to RP as Abathur with me. Same results for me, it works for the first couple responses, but very quickly regresses to the mean. I tried writing out a prose/voice guide, I didn’t need it to use actual Abathur quotes, but it never seems to stick. I think there’s a thing the caveman mode does that re-injects the instructions to keep it in line, I haven’t gotten around to experimenting with anything yet though.
That is a useful hint, thanks!
I make Abathur rephrase multi-line comments, because every LLM out there enjoys peppering code with them.
A successful recconect resets the counter. Efficient.
Doesn't always catch that distinctive style, but at least all the fluff is gone.
This Zerg is a software engineer at heart and has no tolerance for nonsense.
Plot twist: GP _is_ you.
> the doctors start saying things like, “Have you considered using a surrogate?” and, “Sure, the last 5 embryo transfers failed, but what if we try a 6th and cross our fingers really hard?”
To be fair, this level of reasoning is on par with free tier chatgpt.
To me this story is less about a triumph of AI and more about a failure of the healthcare system.
Gemini 3 correctly diagnosed the cause of my skin issues that a real dermatologist failed to diagnose, despite multiple visits. I followed its advice and now the issues are completely gone.
What was your issue if you don't mind me asking? It's really hard to verify efficacy otherwise.
Allergy/sensitivity to specific ingredients in some products I used. The products were specifically marked as "dermatologically tested" and intended for sensitive skin. My doctor was surprised, since he uses the same cream, recommends it to patients, and never saw any issues. Yet Gemini took one look at the ingredients list and pointed out the potential offender that can cause rashes right away.
The more time I spend with the US medical system, the more I realize that nobody cares about your health as much as you do, and you really do need to learn how to engage with it, understand what's happening, and then decode how the system works and actively advocate for your own care.
It doesn't matter how great your doctors are, you're still only going to get an hour or so of their time per month if you are REALLY lucky.
Navigating this is really hard. The more AI assistance I can get the better.
> nobody cares about your health as much as you do
This is true about not just health, but literally everything. That's one reason AI is a big deal even when it's subhuman in performance. It empowers the individual to take care of themself (with as much "effort" as they'd like) rather than relying on another human.
Some examples:
- your health.
- your home.
- your children's education.
- your car.
- your finances.
- your (and your family's) safety.
- your elderly parents' welfare.
Most people don’t understand this fact. Some of us are lucky enough to have friends or family that can help us as a health advocate. There are even people who work as health advocates as a profession.
You are the only person you can truly depend on when it comes to advocating for yourself and your health. But if you know that you are not properly equipped to do that job, then you need to get yourself a good health advocate. It’s like hiring a lawyer, but for your health.
Sadly, yes — that is what our healthcare system has devolved into.
> the more I realize that nobody cares about your health as much as you do
This is going to be true in all medical systems, and really in all systems for all fields.
No one else (other than maybe your immediate family) is going to care as much about your situation as you are. Any doctor, in any system, is going to have multiple patients, and they will care about all of them. You, an individual, is going to care about your own life because it is the most important thing to you, and you aren’t splitting that amongst many different patients.
I got sent this post by a friend yesterday and I read it with my wife (physician). It was interesting although the selective inclusion of details as part of writing a compelling narrative made it difficult to interpret the experience of using AI.
The post seems to portray the fibroid as objectively the cause of the symptoms, although there is no actual evidence of that. My wife tells me that fibroids grow and shrink in size over time, and that larger studies trying to quantify fibroids as a cause of infertility have been inconclusive. A point in time MRI seems to be correlation rather than causation.
It seems the common theme using AI for personal health issues is that post writers (partly due to privacy and length concerns) leave out the gory details like a full timeline of what tests were suggested, what they showed, how many providers were visited, and what treatments were trialled. The timeline is not really clear in this case, but clearly a lot of expensive tests, treatments, and providers happened in the almost four (!) years since they started using AI. Some of those interventions might have happened due to AI advice, and there are clues to this like the mention of possible endometriosis in the biopsy result. Unfortunately, there is just not enough info in the post to know.
That being said, it's clear that AI is helpful for explaining and assisting in advocating for health and luckily there was a positive outcome. Hopefully we can work out the optimal way to incorporate it into the health system.
This story is written like it vindicates AI medicine but really it seems like the human medical system dropped the ball on the MRI. ChatGPT didn't have any brilliant insight, it just (eventually) suggested the same thing a human doctor had already suggested, but this time the author and his wife pushed harder for it to happen.
To use the metaphor from the article, they found the keys in the last place they looked.
>it just (eventually) suggested the same thing a human doctor had already suggested
Of course, but what else would you think an LLM does? That is the insight: sifting through a sea of information and parsing it without the same emotional or other cognitive biases the original human recipient applied to it.
> but this time the author and his wife pushed harder for it to happen
This is the essence of how to make the medical system work for you. Advocate for yourself. AI helps know what to ask, why you should ask, and how to get past the first few layers of dismissals.
In this case it took roughly four years for AI to suggest a test. How should we interpret the utility of AI in this situation?
You don't understand. They were only able to feed in information that they obtained with their human medical team. So the AI was heavily constrained in what it knew.
AI was able to suggest a test for four years and it didn't, and I'm wondering why it didn't.
Again, this seems more like an indictment of the American medical system than kudos for AI.
For the developers here, consider: "So our CI pipeline is super flaky, and it you have to restart it in a variety of non-intuitive ways. LLMs have been a massive breakthrough because the bot can run in an endless loop reading the confusing error messages and retrying the pipeline with different environment variables until it passes!"
The human medical system drops the ball all the time unfortunately. What I don’t understand is how they didn’t see that giant fibroid from ultrasound alone. An MRI shouldn’t have been needed. Then the doctor trying to talk them out of the MRI, I hope they were embarrassed after. So upsetting. My wife’s friend had something similar happen after 2-3 years of unsuccessful fertility treatment, then they finally realized she had endometriosis. Seems like that should have been looked into sooner.
> What I don’t understand is how they didn’t see that giant fibroid from ultrasound alone. An MRI shouldn’t have been needed
This is really kinda the crux of it.
The AI suggested the MRI not because it had some insight into "oh there's something only an MRI can detect going on", it was just a "collect more data" situation.
Frankly, the author of the blog should probably have gotten a second opinion from a different fertility doctor sometime in the intervening time period.
As someone who also went through multiple rounds of IVF with my partner, the number of ultrasounds you get during the process is immense. It's hard to call missing a peach-sized fibroid that many times anything other than gross incompetence.
I think this is where there is simply not enough information in the post. Fibroids can change in size over time. That being said, there are multiple types of fibroids apparently so it may be a type that wouldn't show up on ultrasound.
Doctors don’t need brilliant insight, they need to essentially leverage human medical knowledge against unwellness in human beings. This is confounded by two measures: too many patients per doctor, and too much human medical knowledge.
I mean if "AI medicine via chatbot" gets better results than the human medical system, I'd say that vindicates it..
this is one case. there's no proof here that AI will generally do better than human doctors. consider also that the power it has to convince people to push for better medical treatments can also be used to push for worse ones, or to halt standard treatment entirely, at which point it is much harder for the medical system to intervene as they aren't interacting with doctors.
It feels like moving toward a local maxima. In this example the LLM advice outperformed one doctor who dropped the ball in some ways, but I do not look forward to a world where we make it even more difficult to talk to doctors, and route most medical interactions to LLMs. I expect that in 5 years I'll be futilely shouting "representative" into my phone as my appendix ruptures.
With this sort of advice from LLMs, I think there's a lot of selective recall that's easily overlooked -- where we gloss over all the bad, dead-end lines of advice we seek from the LLM oracle.
There is some evidence that has been collected so far that, taken across large sample sizes, good AI systems are now better at diagnosing certain diseases, etc…. We started here with Mycin and e-Mycin back in the 60s. Hopefully, these systems can now start to be better integrated into modern medicine systems.
However, the real problem is that modern AI systems still hallucinate. And there is not yet any evidence that I’ve seen that AI systems can be trusted to work at these levels of efficiency and correctness at the tactical single-patient level, as opposed to the strategic population level.
I think lots more work needs to be done here.
> Even if you’re paying a gazillion dollars for IVF treatments, you still only get a few precious moments to talk to your doctor every visit. That’s just how the US medical system works. If you want to plumb the depths of what treatments are possible, or ask super detailed and anxiety ridden questions about every single scan from the latest round of imaging, you’re going to want a chatbot by your side.
Ehm, no. I want doctors by my side working in a system that is not meant to be producing gains for the rich investors but to help the sick and sad.
Just imagine what could be done with all those billions poured into datacenters and GPUs, if they were invested in public goods like a decent health system!
Sure, that is what we want, but I am not sure what you expected OP to do? Put his life on hold, skip trying for a baby, and dedicate his life to trying to change a system that people have been fighting to change unsuccessfully for decades?
OP is playing the cards he has been dealt, not wishing that he was playing a completely different game.
I'm not expecting OP to do anything else, but in
> I think there is a whole lot to be worried about with AI development, and I think we should do much more to make it safe. I also think it shouldn’t be lost how it’s helping out regular people in ways that are far more profound than generating trivia questions for drunk people.
I'm seeing a plea of "AI is really helpful, don't take it away", which might be true in a specific case but is supporting AI in general, and I got the feeling that is what OP wanted with their post. In my opinion, most of the money, energy, water and whatever else is getting invested in AI might be better invested elsewhere, with less damage to society and the environment and still the same nice outcomes.
A health system is not a public good, access to a physician is both rivalrous (a doctor can’t see two patients at once without degrading care) and excludable. A public good, by definition, is neither rivalrous nor excludable. That’s not to say that the government shouldn’t provide private goods to citizens, but the term public provision or common welfare would be more appropriate to describe what the government is doing in those cases
Agreed, thanks for the clarification.