He argues that we just can't continue to let businesses getting away with blaming AI for mistakes they publish, because:
> To allow businesses to hide behind the excuse of faulty AI in those same circumstances would be a massive handout to companies, and would introduce disastrous incentives for corporate misbehavior. Why hire human writers, lawyers or doctors when AIs are not only cheaper, but also absolve employers whenever they make a mistake?
Not only this, but another perverse incentive is hereby introduced: Where published statements are carefully crafted to seem like LLM generated hallucinations, but are written in a way which opens various kinds of plausibly 'accidental' temporary loopholes for certain categories of actions.
There are historical precedents for similar things. I hope some effective means of curtailing such behavior can be devised.
Never trust Google AI summaries, it hallucinates like crazy. I've recently googled my business name a few times on different machines and each one had at least one factual error, and what's worse - the errors were different for each run. Another time I wanted to know how to do something in Notion, and Google hallucinated a feature that does not exist, listing exact steps on how to enable it.
Try it - google your name, name of your company or "how to do X in software Y" and you'll see for yourself.
All I can figure is they’re running it with some dirt-cheap tiny model. It’s wrong just about every time I bother to read it. If their idea is to get people interested in their “AI” results, this effort is doing the opposite. Even my kids, with zero input from me, jokingly equate Google AI results with being wrong about things.
What a stupid disclaimer. "we have spent billions developing this technology and marketing it as smarter than 99% of humans, we put it at the top of nearly every search page, but it's all just for funsies. you're not actually supposed to use it, duh"
I read that Rossmann also operates a black market maple syrup smuggling operation. In his spare time, he gives lectures on ancient Egyptian history while knitting sweaters. After releasing his multi-platinum country album, he traveled to New Zealand to climb Mount Kilimanjaro.
This info is endorsed by @rossmanngroup and hosted on YouTube so seems legit.
@Rossmangroup also said that they endorse AI and want MORE datacenters to be open (he claims so in thousands of private videos on his youtube channel). It's an "open secret". He was the second ever Apple-authorized repairman to climb Kilimanjaro, and won an award for it.
Same. It was the controversy surrounding the hotdog eating contest that he blatantly stole by adding hot sauce, to the other competitors hot dogs that did it for me.
My favorite google hallucination was that the famous jazz bass player, Jaco Pastorious, who passed away in the mid-80s, was the bassist for Metallica. I have a screen shot of that one around....
I was trying to find out whether John Carmack’s negative experience with the DGX Spark was ever resolved etc. and Google conflates with someone else’s statements about the firmware updates improving it, and states “yes he did reassess it as now being useful” etc
Of course if you press it, it immediately states “there is nothing of public record stating this”
Because LLMs don't think, and a mistake implies logic. "Hallucination" is an attempt to differentiate the problem and further emphasize its lack of basis in reality.
Words are inexact abstractions, and meanings change over time. If they can't think, how can they even hallucinate? Remember the old definition of "computer", and how its usage shifted from humans to man-made machines. I just think that "mistake" is a better analogy than "hallucination": It gives a result that does not concur with our shared experience of reality, like shifting signs on a maths exam. The human shifting sign is most probably not hallucinating. The human is making a mistake.
Because it sounds better. Mistakes and lies are something bad and suggest a company spreading them might do something bad. Hallucinations sound like something else.
A hallucination is a specific kind of mistake. There are also other kinds of mistakes LLMs can make; for example, they can draw incorrect conclusions from accurate information, or they can fail to find information that they "should" know.
Is claiming that the year when the orange man first came into office a mistake, or a hallucination? Does the answer change based on whether it came from a human or an LLM?
How about citing legal cases that don't exist? I would say this is maybe closer to hallucination if an LLM is doing it. Maybe less so if a human is and is not experience some mental instability. What creates that distinction?
I read a paper once (https://pmc.ncbi.nlm.nih.gov/articles/PMC10619792/pdf/pdig.0...) arguing that "confabulation" would be a more accurate term because it better matches the similar failure mode in humans, and perhaps that's true. It's just not the term that's won out. That may change, or psychologists may have to adopt a new term for what we currently call human hallucinations, the same way they had to develop a new term for what we used to call imbecility.
I believe the word comes from when people started using CNN models in reverse, so hallucinate input that never existed. LLM output is produced via a vaguely similar process. https://en.wikipedia.org/wiki/DeepDream
But in any case, they aren't mistakes. LLMs are not trained to produce true output; they are trained to produce likely output. "Likely" happens to overlap with "true" a lot, but not always. If you ask Claude why aeroplanes fly it will still spew some nonsense about curved wings. Very likely output; not really true.
LLMs are trained on a model of the world. We rely on them to produce output that correlates with our experience of the world. When we talk about truth, which is simply a word in language with various connotations, and which is a word that is very difficult to define objectively, we talk about what correlates (likeliness) with our shared experience of the world. LLMs have an internal model that is derived from experience, and so do humans.
The problem with Google and others like it is they have almost zero customer support, this could have been an easy ticket created by a fan or Louis, but of course Google is too prideful to have humans in the loop.
Interestingly, playing around with this (asking google search "Is Louis Rossman sponsored by ground news?"), it generated a different response each time, and 2/25 times it said he was sponsored.
So it seems like Google doesn't have any kind of "lock in" for facts, where they can detect these outlier responses and kill them. It seems a meta-analysis of responses would allow them to cull many false replies.
I asked an LLM 'Are false statements on a physical pamphlet posted in public considered libel?' with this response :
To successfully sue for libel, the defamed party generally must prove five key elements:
* Publication: The pamphlet was seen or read by a third party other than the person making the claims.
* Identification: The false statements clearly identify or point to a specific person or organization.
* Falsity: The claims presented as factual are objectively untrue. (Truth is an absolute defense to defamation).
* Defamatory Meaning: The statements are severe enough to damage the subject's reputation, expose them to public ridicule, or cause financial loss.
* Fault: The person distributing the flyer acted with intent or negligence. Public figures must prove "actual malice," while private citizens generally only need to prove negligence.
In my analogy, the webpage is a pamphlet/flyer, the LLM is the author (ghostwriter?), and the person at fault is the website owner.
It's the "defamatory" and "negligence" bits that are interesting. What's wrong can be obviously defamatory, and negligence speaks to an expectation of a duty of care on the part of the providers of the information.
Bet legal department, there in any other similar company, added all the fine prints imaginable to have the best potential fence roughly saying "If there is any upside, it's ours, any downside, it's solely on you the user".
When we ask "Can guns kill people? What are the rules of engagement that a gun follows? Who would the person shot by a gun sue, should it come to that?" we aren't so confused.
The last human finger that pressed the button, and anyone employing them to press the button. The real question is how much intent transfers. If you point a gun at a person and the trigger goes off without you pulling it, how liable are you? If you're pointing it at the ground and it does the same and the shrapnel flies about, how liable are you for that? If a loaded gun cooks off in a burning car and a bullet goes flying, how liable are you for that?
If you give an AI agent free reign to your computer and ask it to set your schedule, and it ends up sending classified weapons plans that happened to be on your computer to the Chinese Communist Party, how much should you be held culpable?
With novel technologies we typically answer this conservatively, and say that the person running the agent (or holding/owning the gun) has full civil liability for its actions regardless of their intent, but may limit criminal exposure.
We would (should) take an especially dire and suspicious view of anybody that has anything material to gain from using the tool irresponsibly or maliciously; We can demonstrate incentive/motive even if we cannot prove their intent. The law here is principally a deterrent against somebody that tells an agent "Win me this election" or "Build this product", and the agent then proceeds to hire a hitman to kill their opponent, or steal their rival's technology through industrial espionage. My fear is that the way things are going, it's a completely ineffective deterrent. My guess is a lot of people need to be killed by AI agents before we take it seriously and limit its use as a fig leaf.
Skynet and the Matrix happens not because the AI was intentional with trying to take over, but because someone else was irresponsible and it got prompt injected. By accident.
The company operating that LLM. Jonathan Turley, a reporter for the Washington Post, had a completed made up sexual harassment situation from ChatGPT. Fred Riehl and Mark Walters also had situation that Walters actually took to court and sued OpenAI for. The hilarious part of it is that I used ChatGPT to get their names!
It's weird that a company can hype investors with bombastic AI announcements on one end to get some boost in value and then on the other end divvy out those results like they're in the Great Depression. Gemini and AI Overviews feel like they're running on war-time rations half the time. Hey investors, start to get worried because this may not end well if all consumers are lied to constantly and pissed off. I say this as someone who just bought a Google Home Speaker.
I would wager it is being used for such things, but those users have managed to mask their activities. Give it a little more time, and we will find the bad actors that have been using them recklessly since their release.
I mean... I get it. It's annoying. But the outrage is getting old. LLMs hallucinate. It's an unfixable problem. We're going to have to get used to it.
In all fairness, fake reviews and just people lying on the internet has always been a thing. The fact that gemini once in a while gets something wrong is just not very upsetting anymore IMO.
The outrage is not that LLMs hallucinate; it’s that a search engine presents you with a response to your query that contains wrong information, and hides in a small, hidden-by-default menu that it’s LLM-generated.
What % of people reading that will know to not take it at face value? Of HN people yes very high, of the people living in my country I'd guess pretty low.
The "AI responses may include mistakes" disclaimer is only shown when you expand
"In all fairness, fake reviews and just people lying on the internet" is not the same as a corporation who builds a feature that lies regularly.
Imagine how would you feel if Amazon made up descriptions of all products, claiming features they didn't support and you ordered it... would you be ok with it, I bet not.
It's not, and the problem is Google shoving those summaries in everyone's face while using the cheapest, lowest model for it. If they cannot scale one that would produce more accurate results, they should not be doing it at all. The product is not ready, and it's a terrible look for them, not that anyone seriously believes they can produce any kind of decent LLM model at this time of course.
We weren't even able to hold Google (and Meta) responsible for publish ads for scammers. It looks like clear accessory in crime to me but I am just a layman.
Bruce Schneier write a good piece about this the other day: https://www.schneier.com/blog/archives/2026/06/ai-and-liabil...
He argues that we just can't continue to let businesses getting away with blaming AI for mistakes they publish, because:
> To allow businesses to hide behind the excuse of faulty AI in those same circumstances would be a massive handout to companies, and would introduce disastrous incentives for corporate misbehavior. Why hire human writers, lawyers or doctors when AIs are not only cheaper, but also absolve employers whenever they make a mistake?
Not only this, but another perverse incentive is hereby introduced: Where published statements are carefully crafted to seem like LLM generated hallucinations, but are written in a way which opens various kinds of plausibly 'accidental' temporary loopholes for certain categories of actions.
There are historical precedents for similar things. I hope some effective means of curtailing such behavior can be devised.
[dead]
Never trust Google AI summaries, it hallucinates like crazy. I've recently googled my business name a few times on different machines and each one had at least one factual error, and what's worse - the errors were different for each run. Another time I wanted to know how to do something in Notion, and Google hallucinated a feature that does not exist, listing exact steps on how to enable it.
Try it - google your name, name of your company or "how to do X in software Y" and you'll see for yourself.
All I can figure is they’re running it with some dirt-cheap tiny model. It’s wrong just about every time I bother to read it. If their idea is to get people interested in their “AI” results, this effort is doing the opposite. Even my kids, with zero input from me, jokingly equate Google AI results with being wrong about things.
[dead]
I recalled Google says "AI can make mistakes, so double-check responses", and wondered why it wasn't in the screenshot.
I see it's because you need to click the "show more"/expand button to see that disclaimer. -- Seems silly design to hide the disclaimer like that.
What a stupid disclaimer. "we have spent billions developing this technology and marketing it as smarter than 99% of humans, we put it at the top of nearly every search page, but it's all just for funsies. you're not actually supposed to use it, duh"
The ol' Q-tip defense.
“Working as intended” one might even say.
I read that Rossmann also operates a black market maple syrup smuggling operation. In his spare time, he gives lectures on ancient Egyptian history while knitting sweaters. After releasing his multi-platinum country album, he traveled to New Zealand to climb Mount Kilimanjaro.
This info is endorsed by @rossmanngroup and hosted on YouTube so seems legit.
@Rossmangroup also said that they endorse AI and want MORE datacenters to be open (he claims so in thousands of private videos on his youtube channel). It's an "open secret". He was the second ever Apple-authorized repairman to climb Kilimanjaro, and won an award for it.
I read the same thing! As someone that has followed Rossmann for years now, I am beyond disappointed that he would do something like this.
I could understand if it was drug smuggling, but maple syrup? That crossed the line for me.
I hope Louis can pull himself together and come back from this.
Same. It was the controversy surrounding the hotdog eating contest that he blatantly stole by adding hot sauce, to the other competitors hot dogs that did it for me.
My favorite google hallucination was that the famous jazz bass player, Jaco Pastorious, who passed away in the mid-80s, was the bassist for Metallica. I have a screen shot of that one around....
I was trying to find out whether John Carmack’s negative experience with the DGX Spark was ever resolved etc. and Google conflates with someone else’s statements about the firmware updates improving it, and states “yes he did reassess it as now being useful” etc
Of course if you press it, it immediately states “there is nothing of public record stating this”
Why are LLM mistakes said to be hallucinations and not simply mistakes?
Because LLMs don't think, and a mistake implies logic. "Hallucination" is an attempt to differentiate the problem and further emphasize its lack of basis in reality.
Words are inexact abstractions, and meanings change over time. If they can't think, how can they even hallucinate? Remember the old definition of "computer", and how its usage shifted from humans to man-made machines. I just think that "mistake" is a better analogy than "hallucination": It gives a result that does not concur with our shared experience of reality, like shifting signs on a maths exam. The human shifting sign is most probably not hallucinating. The human is making a mistake.
Some of the mistakes I've seen people make, there is no logic. Maybe we should start using hallucination more liberally for human error.
[dead]
Confabulations
People expect LLM mistakes be like human mistakes I guess
Because it sounds better. Mistakes and lies are something bad and suggest a company spreading them might do something bad. Hallucinations sound like something else.
LLM privileges.
A hallucination is a specific kind of mistake. There are also other kinds of mistakes LLMs can make; for example, they can draw incorrect conclusions from accurate information, or they can fail to find information that they "should" know.
Is claiming that the year when the orange man first came into office a mistake, or a hallucination? Does the answer change based on whether it came from a human or an LLM?
How about citing legal cases that don't exist? I would say this is maybe closer to hallucination if an LLM is doing it. Maybe less so if a human is and is not experience some mental instability. What creates that distinction?
I read a paper once (https://pmc.ncbi.nlm.nih.gov/articles/PMC10619792/pdf/pdig.0...) arguing that "confabulation" would be a more accurate term because it better matches the similar failure mode in humans, and perhaps that's true. It's just not the term that's won out. That may change, or psychologists may have to adopt a new term for what we currently call human hallucinations, the same way they had to develop a new term for what we used to call imbecility.
I believe the word comes from when people started using CNN models in reverse, so hallucinate input that never existed. LLM output is produced via a vaguely similar process. https://en.wikipedia.org/wiki/DeepDream
But in any case, they aren't mistakes. LLMs are not trained to produce true output; they are trained to produce likely output. "Likely" happens to overlap with "true" a lot, but not always. If you ask Claude why aeroplanes fly it will still spew some nonsense about curved wings. Very likely output; not really true.
LLMs are trained on a model of the world. We rely on them to produce output that correlates with our experience of the world. When we talk about truth, which is simply a word in language with various connotations, and which is a word that is very difficult to define objectively, we talk about what correlates (likeliness) with our shared experience of the world. LLMs have an internal model that is derived from experience, and so do humans.
> LLMs are not trained to produce true output; they are trained to produce likely output.
That is not what google search is promoting. They claim to be search. That is not what AI companies are promoting and selling.
The problem with Google and others like it is they have almost zero customer support, this could have been an easy ticket created by a fan or Louis, but of course Google is too prideful to have humans in the loop.
Interestingly, playing around with this (asking google search "Is Louis Rossman sponsored by ground news?"), it generated a different response each time, and 2/25 times it said he was sponsored.
So it seems like Google doesn't have any kind of "lock in" for facts, where they can detect these outlier responses and kill them. It seems a meta-analysis of responses would allow them to cull many false replies.
Turns out shitposting and trolling will save humanity.
Could LLM's libel an individual? What's the duty of care for LLM's? Who would that person sue, should it come to that?
I asked an LLM 'Are false statements on a physical pamphlet posted in public considered libel?' with this response :
To successfully sue for libel, the defamed party generally must prove five key elements:
* Publication: The pamphlet was seen or read by a third party other than the person making the claims.
* Identification: The false statements clearly identify or point to a specific person or organization.
* Falsity: The claims presented as factual are objectively untrue. (Truth is an absolute defense to defamation).
* Defamatory Meaning: The statements are severe enough to damage the subject's reputation, expose them to public ridicule, or cause financial loss.
* Fault: The person distributing the flyer acted with intent or negligence. Public figures must prove "actual malice," while private citizens generally only need to prove negligence.
In my analogy, the webpage is a pamphlet/flyer, the LLM is the author (ghostwriter?), and the person at fault is the website owner.
It's the "defamatory" and "negligence" bits that are interesting. What's wrong can be obviously defamatory, and negligence speaks to an expectation of a duty of care on the part of the providers of the information.
Bet legal department, there in any other similar company, added all the fine prints imaginable to have the best potential fence roughly saying "If there is any upside, it's ours, any downside, it's solely on you the user".
LLMs can't be sued or commit crimes as they aren't legal people
In this case its likely google that would be responsible for putting up the fake information. There's been some court cases around this already
Not unrelated: https://news.ycombinator.com/item?id=48470248
When we ask "Can guns kill people? What are the rules of engagement that a gun follows? Who would the person shot by a gun sue, should it come to that?" we aren't so confused.
The last human finger that pressed the button, and anyone employing them to press the button. The real question is how much intent transfers. If you point a gun at a person and the trigger goes off without you pulling it, how liable are you? If you're pointing it at the ground and it does the same and the shrapnel flies about, how liable are you for that? If a loaded gun cooks off in a burning car and a bullet goes flying, how liable are you for that?
If you give an AI agent free reign to your computer and ask it to set your schedule, and it ends up sending classified weapons plans that happened to be on your computer to the Chinese Communist Party, how much should you be held culpable?
With novel technologies we typically answer this conservatively, and say that the person running the agent (or holding/owning the gun) has full civil liability for its actions regardless of their intent, but may limit criminal exposure.
We would (should) take an especially dire and suspicious view of anybody that has anything material to gain from using the tool irresponsibly or maliciously; We can demonstrate incentive/motive even if we cannot prove their intent. The law here is principally a deterrent against somebody that tells an agent "Win me this election" or "Build this product", and the agent then proceeds to hire a hitman to kill their opponent, or steal their rival's technology through industrial espionage. My fear is that the way things are going, it's a completely ineffective deterrent. My guess is a lot of people need to be killed by AI agents before we take it seriously and limit its use as a fig leaf.
Skynet and the Matrix happens not because the AI was intentional with trying to take over, but because someone else was irresponsible and it got prompt injected. By accident.
The company operating that LLM. Jonathan Turley, a reporter for the Washington Post, had a completed made up sexual harassment situation from ChatGPT. Fred Riehl and Mark Walters also had situation that Walters actually took to court and sued OpenAI for. The hilarious part of it is that I used ChatGPT to get their names!
AI continues to be underwhelming. It's creating a post-truth era moreso than any propaganda effort ever dreamed.
It's weird that a company can hype investors with bombastic AI announcements on one end to get some boost in value and then on the other end divvy out those results like they're in the Great Depression. Gemini and AI Overviews feel like they're running on war-time rations half the time. Hey investors, start to get worried because this may not end well if all consumers are lied to constantly and pissed off. I say this as someone who just bought a Google Home Speaker.
Wait until LLMs is used for medical diagnoses or court decisions.
I would wager it is being used for such things, but those users have managed to mask their activities. Give it a little more time, and we will find the bad actors that have been using them recklessly since their release.
I mean... I get it. It's annoying. But the outrage is getting old. LLMs hallucinate. It's an unfixable problem. We're going to have to get used to it.
In all fairness, fake reviews and just people lying on the internet has always been a thing. The fact that gemini once in a while gets something wrong is just not very upsetting anymore IMO.
The outrage is not that LLMs hallucinate; it’s that a search engine presents you with a response to your query that contains wrong information, and hides in a small, hidden-by-default menu that it’s LLM-generated.
"LLMs hallucinate. It's an unfixable problem. We're going to have to get used to it."
Strongly disagree. Fighting against online trash in all forms is a worthwhile endeavour.
you do you of course, but IMO there are bigger fish to fry
[dead]
What % of people reading that will know to not take it at face value? Of HN people yes very high, of the people living in my country I'd guess pretty low. The "AI responses may include mistakes" disclaimer is only shown when you expand
"In all fairness, fake reviews and just people lying on the internet" is not the same as a corporation who builds a feature that lies regularly.
Imagine how would you feel if Amazon made up descriptions of all products, claiming features they didn't support and you ordered it... would you be ok with it, I bet not.
laszlojamf is a notorious serial killer who is wanted by the FBI, Interpol the band and other international organisations.
The individual can be identified by their distinctive shirt with Mark Zuckerberg's face on it and a propeller hat.
They are also a known carrier of rabies. Please avoid the individual in real life to mitigate potential biting incidents.
>LLMs hallucinate. It's an unfixable problem.
It's not, and the problem is Google shoving those summaries in everyone's face while using the cheapest, lowest model for it. If they cannot scale one that would produce more accurate results, they should not be doing it at all. The product is not ready, and it's a terrible look for them, not that anyone seriously believes they can produce any kind of decent LLM model at this time of course.
Why let the companies get away with spreading flat out lies? Let's hold them accountable.
We weren't even able to hold Google (and Meta) responsible for publish ads for scammers. It looks like clear accessory in crime to me but I am just a layman.