> in part because Google fired two of the authors, Timnit Gebru
I remember being angry about this situation when I first saw it on social media, until I read the details: This person submitted a list of demands to her employer and said that if they weren’t met, she quit. Google wasn’t going to meet her demands so they considered it acceptance of her resignation. There has been a movement trying to debate whether it was a firing or resignation ever since.
The original paper they published gets recirculated every year or two as some landmark history of AI safety, but as other commenters have noted it wasn’t really a great paper nor was it groundbreaking at the time. If not for the controversy surrounding the resignation/firing (depending on your POV), I don’t think it would have been notable.
I think part of it is she had excellent PR skills and a dedicated fan base. I was at Google when she quit, working in ML, and hadn't heard of her until the story broke. I remember there were a large number of Memegen posts about it, but no one I spoke with knew about her, so I assumed it was brigading.
I think she's since since lost a lot of her allure, especially when she didn't change her mind when the facts about the AI water usage changed 1000x
What exactly are the facts about AI water usage? I have trouble separating hysteria from reality but most of what I see still claims water usage is enormous
The hysteria around water usage rests on people not knowing the scale of industrial civilization. First thing to do is compare any estimate of data center water usage with the water usage of almond farming. Or, if you want to focus on individual consumer choices, the water footprint of eating a hamburger.
> Or, if you want to focus on individual consumer choices, the water footprint of eating a hamburger.
To drive this point home: if every American ate exactly one less hamburger per year, it would entirely offset the annual water consumption of all US datacenters (including, therefore, the water footprint of AI).
> if every American ate exactly one less hamburger per year
I fear that you have unwittingly led us further into the inevitable Environmental Hamburger-Cost Continuum.
Wherein we will use EHCC units to inform the hamburger voters about a) the nobility of even the smallest hamburgerian sacrifice, and b) the gentle, loving hand of industrialisation.
I prefer to use the hospital analogy. Locally, the water concerns are a big deal but a lot of people in my community are riled up about the potential need for diesel backup generators because of the noise pollution. They are not wrong and it's good to consider, but they are at this point grasping for reasons that would not (and have not) been concerns for other large footprint projects like hospitals with similar infrastructure needs.
Here in Reno the anti-datacenter crowd has been extending that tolerance to the local casinos and golf courses (of both of which there are many more than we arguably need). If the choice is between running servers to keep the Internet going v. letting poor schmucks get swindled out of their money on the slots, I'm picking the servers over the slots any day.
Breaking it down this way is a great way to minimize the numbers so that it appears reasonable.
See? Middle-Eastern investors are growing alfalfa in the western desert using legal allotments of water! That is so much worse than what we’re doing! Go after them!
They can both be using an egregious amount of water for silly purposes.
The other part of the water debate is also the pollution different systems create. Many data centres went in with the promise of closed-loop systems but changed half-way through construction and couldn’t be stopped.
I think it’s more complicated than, “they’re wrong, it’s just hype.”
Putting things into perspective is not minimising the problem. We literally have to do this to prioritise where our efforts can be useful.
Your argument makes sense if ai datacenters were using something close to like alfafa farming but the difference between them is soo massive it does not make sense.
Reducing pollution is a much better problem to fight for
I am guessing (not asserting) that there is a sort of cap on water used for agriculture. It's possible we've already reached it. (?)
So, on the matter of scale: there likely isn't a cap on water use of these datacenters. Both the heat emission and usage levels for these systems will likely go up unless there is a fundamental technical breakthrough.
On the matter of utility: As a sibling of GP mentioned, the utility of food is clear.
On the matter of polution: I am not remotely read on waste water and contamination due to industrial agriculture. Is this also something where the judgmental scale is tipped in favor of food production vs cooling systems?
what is the utility of alfaalfa? Because as it stands it's essentially just a way to export enormous amounts of water from a place where there's little (California) to a place where there's none (the Arabian peninsula).
Your alfalfa is a red herring. Agriculture encompasses more than a specific crop that we can always legislate against production for export. The data centers will not go away and the utility of the "AI" they are to enable is as of today dubious.
> This is bullshit to be charitable.
What is bullshit is sockpuppets shilling for the very few who have bet the 'farm' on this and want return on their investment, the wellbeing of society be damned.
Everything we do has side effects. Some have more than others; in the case of water, even heavy users of AI have a relatively small AI water footprint compared other things they do.
There are ideological perspectives--another comment in this chain declares that anything that an LLM does has zero value, purely by virtue of being from an LLM. There's not much to discuss there: maybe I find alfalfa useless and LLMs provide a lot of value, maybe you do the opposite. Short of violence, either immediate or delegated, the best way to resolve this conflict is to incorporate costs for resources like water appropriately (accounting for externalities), and let market participants bid to determine who gets access to it.
Datacenter owners are highly likely to find this a reasonable process, because they believe that what they are investing in and operating will provide a great deal of value on the market. And other industries (like your alfalfa) are far more likely to throw a fit at this process.
Or if you think alfalfa farms have fundamental, deep-seated water rights that don't extend to data centers, then data center operators can just purchase water from alfalfa farmers, and everyone involved will eagerly make that trade.
Well the other thing is that agricultural uses are generally on a different scale and use different sources. And at least you can eat almonds and alfalfa. Probably more useful than asking an LLM to autocomplete your homework. Depending on your needs.
Even amongst municipal water usage, it seems hard to figure out how much DC usage factors in to other uses because often they’re not required, and therefore don’t, disclose their usage.
All I’m saying is that it’s not a closed case and not everyone complaining about DC water usage is a Russian propaganda bot.
Nice false dichotomy you got there there, might as well calculate the entire water usage for a a single GPU in the supply chain too. Something tells me one is extremely worse than the other when you account for all the water that's used in a single supply chain for high end electronics, but if you want to plop the measuring stick where ever along the whole pony show that makes you look better people will notice.
Also to compare growing food with the totally optional, not useful in the slightest, LLMs that somehow demand local populaces bend to their will for reasons that never seem to benefit them is just bonkers level of self-blinding when it comes to populations absolutely despising big tech, big tech leadership, and big tech practices.
This mania might finally cause the software industry to become a highly regulated with licenses similar to that of other engineering disciplines due to amount of optional destruction they have decided to unleash upon on the planet in such a short time frame.
let's compare ai water usage with something even more useless, water leaking out of pipes. the United States currently loses about 2 trillion gallons of water annually to leaking pipes. this is in comparison to around 230 billion gallons used by data centers.
we're literally dumping several times the amount of water used by data centers onto the ground for no benefit at all. oddly enough I haven't seen any protests about this despite how concerned everyone is about water usage.
How do you know this doesn't suffer from Gell-Mann Amnesia? The first version had so many glaring errors that have been "corrected" (removed), and I don't have the energy to comb through this one.
I am highly skeptical of layperson debunking like this.
The thing that is off putting about how he uses rhetoric is that it feels like and-you deflection (tu quoque).
> Claim that a data center is using 1000x as much water as a city of 88,000 people, where it’s actually using about 0.22x as much water as the city, and only 3% of the municipal water system the city relies on. She’s off by a factor of 4500. This is the single largest error in any popular book that I’ve found on my own, and to my knowledge I’m the first person to notice it.
Yeah, she might be wrong. But a data center also using 1/5th of the water consumption of an 88k person city should still be what are debating. We also have a base rate fallacy, we don't know know how well or how poorly they are using water. Nearly all criticisms of AI data center criticisms boil (no pun intended) down to yeah, but what about almonds or rice or xyz. That isn't a healthy way to adjust how we talk about data center water consumption.
It is a classic debate tactic. Someone makes an argument, then buttresses that argument with a number. You attack the number, pretending that you didn't just correct the number, but also invalidate the original argument. We shouldn't be using these tactics to talk about the tragedy of the commons.
I worked for a hyperscaler, I poked around a bit about water usage both internally and externally and it wasn't good. There was little to no thought other than, "we can pay X to have water delivered, doesn't matter if it sourced responsibly." (to glibly paraphrase, company policy is to never write the honest part down)
Look at how hard Google fought to not have water data released in The Dalles Oregon for their DC there. Many DCs are supplied by water that is meant for humans, sourced from aquifers that took hundreds of thousands of years to fill, that are being depleted faster than they are filled, already.
I think AI is powerful tool, but we still can't give DC expansion a pass.
I'd love to see DCs scaled with renewables, they would throttle down and run slower at night. The abstraction that power is available at whatever rate you want it is an expensive myth at scale.
I defer plugging in my electric car until 10am so I know that my neighbors solar arrays are charging my car.
In Seattle, every watt saved is water saved since we are blessed with hydro.
> But a data center also using 1/5th of the water consumption of an 88k person city should still be what are debating.
Why, when that's such a minuscule amount of water in the grand scheme of things? Why focus our energy there? Why not spend that energy on the myriad many-orders-of-magnitude-worse offenders?
So I'm not sure if it's the first version of this doc, but I ran the first version of it that's on Wayback Machine[1] with the current version to see what the differences are.
Most of the changes seem to be because Masley found one counter-example (Newton County, Georgia) where AI datacenters do seem to be increasing water costs; the only deletions AFAICS is toning down language where Masley used to say "there are no examples" to "there is one counter-example". I don't see any other major corrections that have been removed.
What is interesting about it for me is: why would someone working on AI ethics choose to work for Google at all?
Did she really think Google cares about ethics? Such positions seem purely performative, we all know that ethics go out the window first to make room for more profits.
I missed that thread when it came out, it's really a wild read. The difference between how people describe her vs how she's normally portrayed in the media is really startling
People generally get fed the stories they already believe to be true
So that's why "Brave minority woman unfairly fired by evil AI corporation" sells better than "Self-entitled minority woman is terrible bully to colleagues"
True but also... she wasn't a software engineer putting code in production nor a researcher working no the fundamentals of machine learning negotiating a raise.
She was part of the "Ethical Artificial Intelligence Team" of what was then, and still is now, one of the corporations World wide spending the largest amount of resources precisely on using AI commercially.
I'm saying the paper itself wasn't a bombshell or even that noteworthy. The reason it got PR and continues to come up was because the authors manufactured this self-inflicted drama around it, not because it was leaking secret revelations that harmed the company.
Her demands included wanting to know the identities of anyone who wanted to comment on her paper, after she had a history of going after people publicly. That's enough right there, nobody should tolerate toxic behavior regardless of whether you agree with the politics.
Meanwhile, the paper has 2 points of criticism towards AI. 1 is a bunch of carbon consumption complaints assuming NVIDIA cards with coal-fired power, while a lot of effort at contemporary Google went towards getting TPUs running on green power. I suspect this was what people wanted to object to, a lot of effort went into those green power projects and she was just denying it. The complaint seems prophetic now but it was not true about Google then.
The other criticism was about which language the LLMs use, they average the input data of normal humans instead of talking the way the paper author thinks they should talk. The phrase "women doctors" is called out as problematic. I'm less inclined to think people objected strongly to this given the zeitgeist at the time, it was probably people who worked on the green energy projects and were pissed off that their contributions were ignored, but still, nobody elected her Queen of English, she can have her opinions but she's not a victim for not having them adopted by everyone.
One of the key points of that paper is that the body of written works is biased and those biases will be amplified by compressing that body into LLMs, with outlier data, with low coincidence rate, being suppressed as unreliable - data that is structurally dissimilar to the bulk is noise. Similarly to how PageRank suppressed nodes with low number of edges, probably contributing significantly to the homogenous corporate mall-internet we enjoy today.
I think this is completely misleading. If your employer asks to redact your paper because it ignores relevant research, naturally you want to know what research they are talking about and on what grounds, and also which researches reviewed the paper (probably assuming there was none, review process was used as an excuse to silence criticism, G has done it many times).
(BTW, quite bold to say input data from Reddit and 4Chan is how “normal” people speak. There is a lot of language in the training data of any model you really do not wish your application to use ever.)
Ugh. Ok if you're going to push one-sided propaganda I'll push the other side.
Google forced its researchers to retract an already submitted paper because it undermined its strategic and commercial story around large language models. The "we just accepted her resignation" is just a lie. Google made harsh demands with opaque reviewers that made vague objections, and then Jeff Dean moved very quickly to get rid of Gebru. Other Google researchers reported that they usually got to work through objections, Gebru got no such opportunity. Google showed that AI labs will not tolerate internal research that seriously criticizes technology central to its business.
Dean pulled a sweet Dungeon Master move in "accepting her resignation." She should have made them fire her, esp for ostensibly doing the job she was hired to do.
I think a better summary is that google gave someone the responsibility to ensure AI was developed and used ethically, and didn't give them the power to execute that responsibility.
Apparently google did not even give them the freedom to come to their own conclusions.
I just skimmed through the paper again and what it says seems to hold up well.
The paper is heritical in that it suggests slowing down, using carefully chosen less biased data, and understanding how it all works.
Basically challenging the AI bitter lesson.
I can see how that would meet with a lot of resistance from people who have gone all in on the bitter lesson.
I'm surprised that people still take Gebru seriously. She is a disgrace to the community because she always, I mean literally always, attacks her critics by motives. You think bias is a data problem? You're a bigot (See her dispute with LeCun). You disagree with my assessment on an ML model? You are white male oppressor (her attacking a Google's SVP). Oh, did I mention that she even said that some loss functions are more racists than others on X?
Gebru is not a researcher. She is a modern-age Trofim Lysenko, who politicizes everything and wields political correctness as a weapon to purge any dissent.
This is not why she was fired and wouldn't have been a plausible reason in 2020 when it happened.
>Timnit responded with an email requiring that a number of conditions be met in order for her to continue working at Google, including revealing the identities of every person who Megan and I had spoken to and consulted as part of the review of the paper and the exact feedback. Timnit wrote that if we didn’t meet these demands, she would leave Google and work on an end date. We accept and respect her decision to resign from Google.
This is Google's side of it; I think the following is a fair piece of primary-source journalism if you want to go deeper:
She was probably repeating behaviors she learned in academia. These kinds of extremely toxic "don't just apologize for disagreeing with me but also give me the name of every person involved for collective punishment" is a classic move for academic tyrants throwing weight around. The understanding there is that they will then move to cut off every named person from power or access to academic resources. Google did the right thing by protecting their people.
I have watched it happen multiple times that someone from academia joins a research group at a large corporation and finds out to their chagrin that they can't just overtly bully colleagues as easily without tenure.
edit: And it looks like Mitchell immediately left an angry comment about being called a "linguist" despite the article never doing so. Starting to notice a pattern of confrontation, ego, and escalation among some of these people.
> With the octopus thought experiment, I initially had told the story in terms of a dolphin, because dolphins clearly are intelligent animals. My co-author on that paper, Alexander Koller, said it should be an octopus, because first of all, the environment that octopuses live in is much more distinct from where people live. It makes the metaphor more vivid, that the octopus is just feeling these pulses in the cable and has no way to look at what the people are looking at.
It's such a tragedy that they're also extremely solitary animals and die shortly after reproducing the first (and only) time.
Almost all other particularly intelligent animals seem to be gregarious, and it's easy to conclude that a social lifestyle tends to select for more intelligence, a sophisticated theory of mind, and so on (I like to think that that's exactly what was responsible for a runaway intelligence explosion in humans). But in the case of cephalopods, there's something else that has been applying selection pressure towards exceptional intelligence.
Is it a new species though? Or just a living fossil that exhibits behaviors that were bread out of all other octopi species?
It's hard to imagine how high intelligence could be selected for without social component that provides the individuals a lot of pressure to outsmart each other.
Intelligence is plenty adaptive when you're just trying to outsmart the environment.
The notion that this is a "living fossil" is... pretty wild. It's genus octopus. Unless it's wildly misclassified (doubtful), it has a recent-ish common ancestor with most other common octopuses, which all have normal octopus behavior. You find yourself in need of extraordinary evidence.
> Intelligence is plenty adaptive when you're just trying to outsmart the environment.
I think there's very little evidence for that. Environment is just very slow when compared to outsmarting members of your own species that you tightly share the living space with. I don't think you appreciate how abnormal is high intelligence. How pathological conditions must have been to trigger development of something that bizarre and costly. This must have been run-away self-reinforcing feedback loop. We know of many such loops between predators and prey and none of them look strong and fast enough. Where we find intelligence, it's almost always tightly connected with social behaviors. It would be exceedingly strange if there was a true exception.
> The notion that this is a "living fossil" is... pretty wild. It's genus octopus.
I didn't mean that literally. What I meant was that the common ancestor of this octopus species and others might have been more like it, social, not like the rest. Maybe this species preserved of what normal octopus behavior used to be. If the solitary life became more evolutionarily favorable after the development of intelligence most descendant species could have switched.
> You find yourself in need of extraordinary evidence.
Yeah, maybe at some point we'll discover a huge collection of beaks in some under-ocean sediment and it will become obvious. Ocean floor is not really that explored.
The evidence is literally every other evolved form of intelligence. Including, despite your speculation, octopuses. Do recall "the environment" also includes other intelligent agents an animal needs to compete with. Does that help you understand? A runaway process might be required for human level intelligence but clearly not in general.
Basically all octopuses are solitary and die after breeding. LPSOs are nested in one group of octopuses. You're proposing that one group of octopuses developed social behavior and multiple breeding, then had a bunch of descendants who went back to exactly the normal octopus behavior. Hopefully it's obvious why this fails Occam's razor.
> > I think there's very little evidence for that.
> The evidence is literally every other evolved form of intelligence.
Not really, even harsh environments don't really select for intelligence. Neither does predator-prey dynamics. There are very few significantly intelligent species, and except for octopuses they are all social.
> Do recall "the environment" also includes other intelligent agents an animal needs to compete with.
If you include members of its own species in "the environment" we don't have any thing to discuss, because I fully agree that "the environment" understood as anything external to the individual can trigger the development of intelligence. Because it's trivial to observe that it did.
My claim is more informative (if true), that for the development of intelligence the environment must include populations on individuals living in close proximity, social, interacting a lot, where the strongest evolutionary pressure on a given species is the species itself. That doesn't seem to be true for most octopus species today. As an exception to the rule, their intelligence is interesting. I personally believe it comes from the same source as every other intelligence. And as the living space for octopuses expanded, they became more solitary. Maybe there's something general about this too. That with sufficiently intelligent species (perhaps if the intelligence can't rise anymore due to physical limitations), the other individuals present such a high danger that the species must spread apart and become more solitary. Maybe humans reached this stage already and spread apart, but as they ran out of Earth, they were forced back into old, more social mode, just now on the planetary scale. There is evidence that humans in Europe lived in way smaller groups and used to be way more murderous and cannibalistic not that long ago.
> Basically all octopuses are solitary and die after breeding.
That fits. There's no point in extending the life beyond reproduction if your kids are gona kill you anyways.
> You're proposing that one group of octopuses developed social behavior and multiple breeding, then had a bunch of descendants who went back to exactly the normal octopus behavior.
I'm not proposing that at all. What I'm proposing that all ancestors of octopuses were social and could breed multiple times. However, because of the proximity they became their greatest evolutionary pressure for themselves as they got more intelligent. Eventually they spread out due to some environmental change and their social behaviors and multiple breeding atrophied due to danger from other members of theirs own species, that now could just be nearly completely avoided. And that's the case for nearly all descendant species except this special one for some local environmental reason. This species shows us that social octopuses are (and were) a possibility.
What you consider normal octopus behavior is as normal as eyes of a molerat.
That's a fun story, but it doesn't match the evidence at any point. Your fixation on intraspecific competition as the only significant driver of intelligence is leading you far astray.
In Stefan Wul's SF novel "Nyourk", octopi evolve to become the Earth's dominant species, which was quite prescient back in 1957, when almost nobody knew octopi to be intelligent. :)
It's also a bit of a chicken-and-egg problem: if they were raised by their parents like all other more intelligent animals, they wouldn't need to be as intelligent as they are in order to be able to relearn "octopus behaviour" without help from other members of the species.
That's a bit exaggerated - monkeys can't fly, but both dolphins and octopuses can swim. I'm aware octopuses prefer to stay at the bottom, while dolphins have to come to the surface regularly to breathe, but for me it's still the same environment...
Tidal basins and ocean bottoms are very different from the open ocean surface. Much like tree tops and 3 miles in the air are different. Both are up in the air, but still quite different, just like "ocean" but still different.
You know monkeys can swim? Do you know what happens to a monkey 2-6km under the surface? Monkeys can also climb trees, very high. Monkey still no condor, sadly.
The continued use of animal metaphors is doing them a great disservice. Esp as we learn more about animal cognition, on first look, it smacks of human exceptionalism that has littered the historic scientific consensus.
Now if they had said, "Imagine your average American ..." (/s)
OpenAI offered ChatGPT to the world. A large, monied cross-section of the world had yet to throw its capital behind the Large Language Model technology that made the ChatBot possible. While it is fair to see AI development now as a global imposition, OpenAI did not have the agency as a 2022 startup to impose on the scale we see now.
I asked Mistral, and it guestimated that Altman, Thiel, Musk, and Hoffman had like $20.3B together when they founded it. Sound to me that the founding of OpenAI was exactly the point when the monied world threw its dollars behind AI.
I think "imposed" is a pretty fair word. LLMs already affect everyone's lives whether you use them or not and their footprint is still growing. It's like I don't have to buy a car, but cars are absolutely imposed on everyone anyway.
Her language consistently defines LLMs in negative terms like “synthetic text extruder” but she claims she’s not trying to denigrate it. What’s missing for me are similar terms from her about how humans create sentences and thoughts. Judging by the state of the internet humans are quite capable of making shit up to argue their point (see latest Fox News apology). She talks about sycophantic AI but give me a car battery and some cables and I can train a sycophantic human (no I can’t but there are people who can). She’s pretty much a walking counter argument for her own claims.
No you see, she says right there in here "Frequently Unasked Questions" (lol) that she is in fact not at all bitter about her field being effectively solved by someone else, thank you very much! Her preferred approach is in fact alive and well, more important than ever and still very very interesting. One wonders if her PhD students feel the same.
The authors of the paper make some good points (in it and elsewhere), but I've seen precisely nothing that suggests they aren't awful people who resort to playing the victim card, inciting cancel mobs and in general behaving like the worst stereotypes of "leftist academia" in response to any criticism.
I paid a bit of attention to this paper and the phrase 'stochastic parrots' when it came out and i thought this was worth saying and doing at that time. their suggestions about financial and environmental costs are worth studying, their concern about carefully evaluating datasets to feed to the model rather than feeding the entire internet is fully justified. so - to everyone saying this was a bad paper; if you have actually read the paper then please list a few criticisms. all i have seen is "oh this wasn't that good of a paper" or "can't believe how bad this paper was".
My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation.
It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!
Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.
not sure your criticism makes sense though - they did this pre chatgpt. they are talking about the language models of that time. they did not make predictions about the future.
They made a claim about language models in general, not just ones that had been released so far.
The point of the paper, in fact, is that language models are getting "too big", and another approach is needed to make progress, so they were certainly predicting things about later models.
With that said, they talked about "pure" language models, so it is fair to say that they didn't talk about, say, LLMs that are multimodal or that have tool use, which are advances that happened after their paper.
They did use RLHF at the time, at which point it is not a pure probabilistic representation of the training corpora. Bizarrely, RLHF never came up in the paper.
We’ve know that since 1943 when McCulloch-Pitts came up with the first “artificial neuron” definition. And since LLMs are a descendant technology — our assumption should be they’re reasoning in some internal learned logic.
This is what the evidence supports — eg, the “stochastic parrot” crowd never can explain transfer learning. Whereas for the internal reasoning crowd that is easy: removing your top level judgments from a theory still leaves you with useful terms for describing a new theory — eg, removing your judgments about “which animal is this?” but preserving the underlying structure for representing an image in your new judgments, “is this cancer?”
There’s 80 years of reason to think DNNs reason and zero support other than “sTaTs R mAgIc!” to support the stochastic parrot interpretation.
And even when they are: they sure seem to bet against Moore's Law or just the general tendency for things to get better/efficient over time.
It's frankly remarkable how capable the models have become that we can run locally now on a decent laptop.
The same thing happened with image generation. I've had arguments with people that image generators are killing the environment, but I can do it in 20-30 seconds on my GPU. No one bats an eyelash when I play 20-30 minutes or even hours of a video game on my GPU, but the images are burning down the planet.
The contention that there is no grounding because the training data is linguistic and thus can only reference a world model is disproven in "This sentence has five words"- there's real, grounded information about what "five" means within that sentence. While that's a trivial counterexample, I don't know that it's an obvious one (I didn't come up with it myself).
It's not a criticism of the paper itself, but multimodal models came shortly after and provide grounding that is more of the sort the paper is getting at, and it didn't seem like anybody updated on that at all. If multimodal models were still stochastic parrots by the original argument, humans would have to be as well; we don't have any way to ground anything beneath sense data and evolution can't have programmed some innate grounding into us because it didn't either. But (and maybe this is my own misperception) nobody threw in the towel at that point.
I confess I never read the original paper until now, opting to absorb by osmosis instead, and I was quite surprised that they don't really make a deeper case than that. After just a few paragraphs about how they can't be grounded because humans don't express their thoughts directly, it lurches into a page about how they can be biased by training. And they certainly can be, but that has little to say about their stochastic nature- humans are biased as a rule with no exception. (For the record, I only read the Stochastic Parrots section before this reply.)
It's not really a bad paper, but I don't see why it ever carried the esteem it did. Hating on it is like hating on Taylor Swift- she's fine, yes, but for her level of success, one is inclined to question every dumb lyric where others get a pass. (Apologies to Swift fans, substitute a successful artist you don't care for here.)
To add to dwa3592's comment, a sentence is not a self-contained idea. The sentence doesn't include what any of the words in it mean, nor what "this sentence" refers to. The exact same sentence can mean different things depending on the text that surrounds it.
Fair point, and on its own it would be surprising to learn what "five" means from that sentence. But you can extrapolate- across a billion sentences, there will be "the next sentence has five words"s and "this sentence are grammared wrong" and so on. It would not be at all impossible to ground a world model on pure text for that reason. And 'not impossible' is sufficient to invalidate the paper's argument.
>If multimodal models were still stochastic parrots by the original argument, humans would have to be as well; we don't have any way to ground anything beneath sense data
Animals don't passively learn from their perceptions, don't have a separation between training and inference, and don't have a prompt-response execution model. Besides its fundamental biology, the grounding an animal brain has is that when it outputs a motor signal it receives some feedback as to the effects of a signal of that strength. A bird learns to fly because the grounding truth of aerodynamics and gravity consistently respond in a specific way to the flapping of its wings. It doesn't learn by passively replaying thousands of hours of somatosensory recordings of flights.
A multimodal model doesn't have the capacity to do much with a prompt. It has no head to turn to look at an image from a slightly different angle to attempt to gleam more information, doesn't have the capacity to interact with the real thing the image represents in any way, and even if it requests another angle and is given it, it lacks the capacity to learn that new information permanently. A multimodal model knows about images of pipes and facts about pipes, but doesn't know pipes; it doesn't have literally first-hand experience with them.
>evolution can't have programmed some innate grounding into us because it didn't either.
What do you mean? Of course genetics programs ground truths. For example, "forward is the way your face points when your neck is relaxed" and "if you can feel it, then it's part of your body".
>and even if it requests another angle and is given it, it lacks the capacity to learn that new information permanently.
I'd argue this isn't true today, but that the loop for incorporation is long (ie, the next training or finetuning run).
>A multimodal model knows about images of pipes and facts about pipes, but doesn't know pipes; it doesn't have literally first-hand experience with them.
Wouldn't this mean that any human who hasn't seen a pipe in person or interacted with it, similarly doesn't "know" a pipe? Most of us haven't interacted with the vast majority of "things" in the world, yet we're still able to build a model and abstractions for them such that we can reason about them, right?
A bird doesn't learn gravity or aerodynamics, it has no 'sense of physics'. It has sensory neural activity that a scientist can show is tied to these things, but you could, at least in theory, falsify the entire experience of the bird. There is nothing in a bird's brain that directly percieves reality. Colors and sounds and textures and so on are all false primitives that don't exist in nature without us, if that's clarifying. And because evolution acts through organisms, its only access to base reality is through their senses. After thousands of years of effort, we have some pretty good models of what reality actually is, but they're still not 'grounded' in the sense you describe.
>A bird doesn't learn gravity or aerodynamics, it has no 'sense of physics'.
That's not what I said. What I said was that it's physics that provides the ground truth.
>you could, at least in theory, falsify the entire experience of the bird
It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions.
>There is nothing in a bird's brain that directly percieves reality.
Yes, of course there is. Animal sensory organs do not produce false information, nor do they provide the brain an interpretation of what they perceive. The brain may fail to distinguish hallucinations from reality, but those are processes internal to the brain. What it gets from the body is raw physical measurements, and what it sends out is raw motor commands.
A multimodal model doesn't have the same direct access to reality, it just has collections of words and images. It has no capacity to determine the reality of a photograph of a sunset or a CGI render of a dragon. The word "real" is itself meaningless to it; they're both real in that they appear in its training corpus. It lacks the capacity to investigate these stimuli in any way, and can just learn to associate different stimuli in arbitrary ways that appear to make sense to us, but nothing else.
And even then, it's just sampling. Much of what we "see" is a prediction, and there are plenty of optical illusions out there premised on that (plus VR techniques like foveated rendering that take advantage).
>It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions.
The criticism in the paper is of the architecture of LLMs, isn't it? The paper contends
"""Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind.
[...] an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot"""
They're saying that the model cannot learn anything about reality irrespective of training data. Your point is an interesting one, but I think it's distinct. To your point though, this is unfalsifiable from the perspective of the "bird". I can't prove that I'm not a dysfunctional cyborg with false perceptions, which makes me wonder if that's a meaningful distinction.
>Animal sensory organs do not produce false information,
I happen to be in possession of some of these and I think this needs a "usually, under ordinary conditions". (Nitpicky and not critical to my point, but I liked the beginning of this sentence too much to edit it out)
>nor do they provide the brain an interpretation of what they perceive
Whereas this I'd argue is not true at all. My cones interact with wave-particle photons at particular wavelengths. I can't even conceptualize wave-particle duality (though some humans can), but "red" and "blue" are the bread and butter of my visual consciousness. These correspond to firings of my sensory neurons much more than they correspond to anything in reality. If that's not interpretation, what is it/what is interpretation?
>The word "real" is itself meaningless to it; they're both real in that they appear in its training corpus.
I expect we agree that I can show a multimodal model a real and a CGI picture and it can tell me which is which. I can take a CGI dragon to GPT 5 and say "look what I found in my backyard" and it will say "Yeah right". Are you saying this is something only possible thanks to RLHF or other modern techniques? That may be the case, unsure how to test that without access to pretraining-only models. Or would you say my experiment is faulty here and doesn't get to your underlying claim?
On the flipside, I could show some meh drawings of fairies to Arthur Conan Doyle, and he'd say "Whoa, this changes everything". I consider him to be one of the great rational minds of history, but he was unable to pass your test here. (In fairness, he was in his 60s and his senses may have dulled, though his belief in spiritualism at large dates to his prime).
>I can't prove that I'm not a dysfunctional cyborg with false perceptions
It doesn't matter if you are one. If you were an AI researcher and encountered a model that saw things for what they really are you would deem it to be malfunctioning and discard it. Any functional model will always be at least one degree further removed from a raw sensory experience that agrees with yours, than you. Its perception will be invariably filtered through the lens of labeled human output.
>I happen to be in possession of some of these and I think this needs a "usually, under ordinary conditions".
I won't dwell much on what you mean, since you said it's unimportant, but it takes a lot for sensory organs to malfunction, and even when they do, they produce corrupted, not false, information; black blotches, not pink elephants. I assume you were thinking of alcohol or something; drugs that affect perception affect the brain, not the other organs.
An interesting edge case is stuff like entoptic phenomena, but those aren't false perceptions; they're, if you will, hypertrue perceptions (something so true, we would rather not see it).
>but "red" and "blue" are the bread and butter of my visual consciousness. These correspond to firings of my sensory neurons much more than they correspond to anything in reality. If that's not interpretation, what is it/what is interpretation?
"Red" and "blue" are your brain's interpretation of the signals it receives from the eye. Notice how "red" and "blue" are fully abstract words, decoupled from anything physical, whereas if I were to refer to the actual encoding of visual information that passes through the optic nerve, I'd have no choice but to reference physical processes (probably talk about voltage and action potentials; I honestly don't know how the optic nerve works). That's because a retinal cell is a simple transducer. The eye doesn't interpret, it merely converts and encodes. Interpretation is a higher level operation.
You cannot compare the simplicity of the mechanism of a whole eye to the indescribable complexity that is between the sentence "roses are red and violets are blue" written on a book, and the raw perception of red roses and blue violets. But a model will only ever be exposed to that distant interpretation, not to roseness or redness.
>I expect we agree that I can show a multimodal model a real and a CGI picture and it can tell me which is which.
Of course, but obviously that's not something it knows inherently. There's nothing about the image intrinsically that says it's fake; someone has to label it such that the model can associate it with unreality (according to our own parameters). That's not how an animal works. An animal assumes what its senses perceive is real and can distinguish its own thoughts from its sensory input.
>On the flipside, I could show some meh drawings of fairies to Arthur Conan Doyle, and he'd say "Whoa, this changes everything". I consider him to be one of the great rational minds of history, but he was unable to pass your test here.
That's a slight equivocation. He would not have mistaken the drawings of fairies for raw perceptions of fairies, he would have simply been swayed by the rhetorical strength of the testimony implicit in the drawing (and perhaps an explicit one that accompanied it). If you want to make a true parallel to a multimodal model you'd have to compare a CGI fairy and a photograph of someone holding a drawing of a fairy. The CGI is as raw to the model as the signals passing through your optic nerves right now, but the drawing is one level of abstraction further away.
>>The contention that there is no grounding because the training data is linguistic and thus can only reference a world model is disproven in "This sentence has five words"- there's real, grounded information about what "five" means within that sentence.
did you think this through?
imagine the sentence was "This sentence has four words", now extrapolate that to all the shit that can exist in a dataset and train a model on that dataset - do you know what will happen? - go ahead and think it through.
I don't think this tone is at all justified. If you think otherwise, I do ask that you point out where I went too far in a comment that I feared was overburdened by caveats and admissions of my own human flaws.
"This sentence has five words" is going to appear far more often than "This sentence has four words". This is the entire premise of LLMs working at all, stochastic parrots or otherwise.
you are right, i was more curt than i should have been. apologies. but you helped prove my point:
>>"This sentence has five words" is going to appear far more often than "This sentence has four words".
it's not about this at all. your point is about data quality. you need to take a step back. the point is that if you trained a language model just on this data set which has sentences akin to "this sentence has two words" - the model is going to learn that. this shows that the language modeling itself doesn't truly provide an understanding of the real world. you can train a language model with the most advanced technology on shitty data and the model will start providing shitty outputs confidently - the model will never say "hey there is something wrong with the data i am trained on". thats what 'understanding of the world' meant in that paper.
If someone substituted all of your sensory inputs for something else for your entire life, how would you notice? If you wore contacts from birth that made the sky red and earbuds that censored when people said it was blue, on what basis would you realize that was wrong? I don't see what this says about the architecture of your brain, and I don't think it's the point being made in the paper. That the training data must statistically connect to reality in order for the model to model reality doesn't seem that important.
>>If someone substituted all of your sensory inputs for something else for your entire life, how would you notice?
I don't know how or if I will notice. That's the biology, chemistry and physics of the brain that I don't know. I hope someone is looking into it. But this does not mean in any way that LLMs are similar to our brains!!!!!! WE DON'T KNOW HOW OUR BRAINS WORK. So going back to language modeling - language models were stochastic in nature when that paper was written, they still are albeit we are trying to make them as deterministic as possible.
I'm quite sure that LLMs are not one-to-one with our brains. It would at least be surprising if instrumental convergence was that powerful, and, even in that case, they're still not getting hungry or sleepy or having dreams, etc. At best they'd match up with a smaller region of our brains, and I'd take 9:1 odds against a close match.
But the paper argues a much stronger point than "LLMs are not similar to human brains": it claims LLMs cannot engage with meaning because they model human language instead of sensory input or human thought.
I also don't think human brains are particularly deterministic.
The authors were wrong about their core thesis and are now lying about it. That's the only criticism needed. They said, quote:
> LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form
... which is presented as unarguable fact, yet is untrue. It was obviously wrong at the time it was written and it's been proven wrong in many ways since. Worse is that they're still at it. In the article she's saying:
> Q: What are the most common misconceptions about the “stochastic parrots” metaphor? Bender: I think one of the biggest ones is, “Bender says AI is a stochastic parrot.”
Her name is on a paper titled "On the danger of stochastic parrots". It has a section titled "Stochastic parrots" and in section 6.1 it says:
> An LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
She did say that, as clear as day. Now she's trying to rewrite history. Ugly behavior.
No, they were correct. In fact an LLM stitches together stuff it observed in its training data. That scales up way better than a lot of us expected, but it's still correct.
If you train it on lots of working code, then it's useful for coding. If you trained it primarily on non-working code it would produce nonsense.
It's not correct. Please read some more research papers, this isn't what people working in AI believe at all. You can prove with experiments that different human languages get translated to the same abstract conceptual space in the middle layers, for example. It's why interpretability is so difficult.
The claim is odd in another way: you can train a person on non-working code and they'll produce nonsense. That doesn't mean people are just stitching together words they've previously seen.
They do not "stitch together" anything. Neither on a technical level, nor a philosophical one. It "scales better than you expected" because your mental model is wrong.
And, not to insult you, but it's quite obviously wrong. As a mental model it fails to explain basic capabilities. How can an LLM follow elaborate instructions? How can it respond appropriately to user input, when the user input doesn't match any previously seen text? Hell - how does it even balance parentheses? There is no way to explain any of this without conceding that the LLM has semantic understanding. It knows that this comes after that, but "this" and "that" can be at an arbitrary level of abstraction.
Sure - they generate text "like" text they've seen before. That "like" does a ton of heavy lifting.
Look, some folks are more impassioned about this stuff than I am. Maybe that's a good thing. But LLMs do in fact just try to predict the next token, using a very big training set. They're very impressive (at tasks the training set prepares them for). But that's how they work.
It requires some level of semantic understanding, like what a paren is and what it means to balance them.
The issue in this discussion is that "predict the next token" is a problematically reductive description of what's going on. It's like saying compilers are programs that emit bytes or that humans are mammals that make sounds. It's not strictly false but it's not capturing the depth of what's happening either.
A simple way to see this is to ask: predicting the next token of what? The obvious answer - predicting the next token that would be found in the training set - isn't correct. If that's what it were doing then it would yield no prediction or random predictions for any prefix not found in that training set, but it isn't what happens. We see generalization and reasoning. They can answer questions never asked before. And once post-training kicks in the question of what it's predicting becomes even harder. It becomes more like predicting what this specific AI assistant would say next, which is a circular definition.
In fact LLMs are trained to predict the next token in the training set. Of course sometimes a new text input doesn't match the training set, or it matches two or more places in the training set. LLMs use a neural network to interpolate, so that's fine. Please look this up if you have any doubts.
Ok. Now. I think you're adding something to the description above. Maybe what you're describing is something "emergent," or maybe it's basically just word vectors that were built in on purpose. You may be adding something correct, or something incorrect. Fine.
But it's not reasonable to say that the "reductive" description above is a "lie". It's not. It's more like a recipe. If you look at correct instructions for making steak, and you call the author a "liar" then you are missing something important.
"interpolate" in what vector space, pray tell? What does "interpolate" even mean, when I prompt it "write me a story about a sentient banana in the style of Hemingway and oh make it a commentary on class consciousness"? You can't assemble such a thing by cutting and pasting pieces of other text. That kind of "interpolation" has to happen at the semantic level - ipso facto, there is a semantic level.
Not to mention that no, they don't predict the next token in the training set. Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?
LLMs aren't even large enough to contain their training data - not even remotely close. It can't "stitch together things it saw" because it doesn't remember them. It only remembers the ideas used to construct them. The learned abstraction is the entire point of the exercise. LLMs would be useless if they were overfit the way you say they are.
>>Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?
you are grossly negligent of LLMs are created. I would highly recommend reading about post training, RLHF, alignment etc. First pass of training is literally "predict the next token". That's it. The first pass is also known as pre training. There's a shit ton of work (instructions, tool use, and reasoning etc) that's done afterwards because the pre-trained is model is useless. if you have some free time, I'd recommend doing this course - https://www.deeplearning.ai/courses/post-training-of-llms
I read your comment multiple times and sorry to say none of your criticisms make any logical sense in any shape or form whatsoever.
>> > LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form
did you read the paper carefully??? this line is directly cited from another paper (pay attention - it's from july 2020) . the full line is -
LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form [ 14 ]
the paper its cited from is titled - Climbing towards NLU:
On Meaning, Form, and Understanding in the Age of Data
>>>She did say that, as clear as day. Now she's trying to rewrite history. Ugly behavior.
did you read the article after that line or was there a shortage of attention span??
> this line is directly cited from another paper (pay attention - it's from july 2020)
It's her own paper, she's citing herself. And the full sentence is "As we discuss in §5, LMs are not performing..." so she's actually just teeing up the same claim she's made previously for further discussion. Why are you trying to claim I'm misrepresenting her words?
> did you read the article after that line or was there a shortage of attention span??
I did. It's more of the same, so there's nothing to say about it.
My criticism centers on the part of the paper they chose for their title, the “stochastic parrot” metaphor. And my criticism is that if you observe Claude code with opus 4.8 working through an entirely novel problem that nobody has ever worked on before and which certainly wasn’t in its training data, the choice to even metaphorically call them stochastic parrots turned out to be egregiously wrong.
And secondarily, and maybe only partially the authors’ fault, is the enormous tidal wave of morons that this paper minted who plague us with their misunderstandings to this day.
oh boy! the lack of critical thinking here is staggering.
>>And my criticism is that if you observe Claude code with opus 4.8 working through an entirely novel problem that nobody has ever worked on before and which certainly wasn’t in its training data, the choice to even metaphorically call them stochastic parrots turned out to be egregiously wrong.
First of all, for your own benefit - Claude code stopped showing the real reasoning. It only shows a summarized version of it now. So don't ever ask someone to observe the model. Second, do you know when this paper came out? do you know when opus 4.8 came out??? how do you know what is novel? did opus 4.8 tell you it was novel? how do you know no one has worked on it before?
I strongly suspect my critical thinking skills are better than yours.
For example, note my use of the phrase “turned out” and then consider whether your main point about the release date of plus 4.8 vs the paper makes any sense at all?
> how do you know what is novel?
Because I work on novel ASICs that were created by my company and have never been used or seen outside of my company?
Personally, I've always read that paper as a political criticism of industry and industrialized research and capitalism. After decades in academic (and industrialized research) I've learned that smart people can write convincing takedowns of things they hate- and those takedowns, due to being well written, often punch above their weight in terms of impact on the community.
I think this paper would have been best split off from the conjoined criticism of environmental effects (which could have been its own paper, but not one published by Google, since their leadership's fundamental beliefs disagree with the paper's environmental impact premise. And the remaining part on text models could have been a bit more focused on the technical issues associated with statistical text processing and meaning, rather than criticism of the power structure that is loosely associated with the current AI push.
She discusses it more in this debate (?) discussion (?): https://www.youtube.com/watch?v=MwfSCCo6jXs . I found it interesting but ultimately thought much less of her and her argument afterwards.
Bender's paper had this to say about stochastic parrots:
"Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."
This was not even a correct criticism in 2021. She is right that, at the time, the pretraining -- where it learns to predict missing words in pre-existing corpuses of text -- is basically a stochastic parrot.
But nowhere in her paper does the term "reinforcement" come up. At the time, this was done mainly through RLHF (reinforcement learning from human feedback) - after the initial training is done, you then tune the model's responses based on human grading. Humans imbue their own meanings into the parameter weights through their judgment.
At this point, they aren't really stochastic parrots anymore, because parameter weights have been shaped beyond the text corpus. It's not purely probabilistic in the sense of using the probabilities of the underlying text sequences. (It still is probabilistic in its output, but that is a pointless claim, because all events in the universe are also probabilistic; it is not enough to merely claim that probability is involved in some way in the outputs.)
RLHF was already in use prior to the paper, and was written about by Christiano in 2017 "Deep reinforcement learning from human preferences," so it's surprising that Bender apparently didn't know about this well-known paper.
RLHF was also, of course, a precursor to a more advanced form of parameter shaping - reinforcement learning with verified rewards, or RLVF, which has driven a lot of the gains in verifiable domains lately. That was not done in 2021 when she wrote the paper. But if you knew about RLHF -- and knew how Alpha Zero worked, with training neural nets on game rollouts -- you could squint and see that it might be useful for language models.
So after being proven to not only having a limited understanding of the field at the time, but also not being able to forecast the field, she's now walking back what she meant by "stochastic parrot," I assume because she believes readers will not read what she wrote. But despite the protests, her original claim was that it is a parrot because the text has no meaning -- a direct quote from the paper, which only really makes sense if training stops at the pretrain.
Even if pre-training was the only training step, it still wouldn't necessarily follow that the only thing the model is doing is stitching words together probabilistically, unless you expand the definition of "probabilistically" to the point that it becomes meaningless. This kind of thinking assumes that design of the training process and the "design" of the artifact that training produces must be similar.
Whether Bender intended it or not, the term has an inherently pejorative sense. "Parroting" is not really indicative of what modern LLMs do. However, when most people bring it up as a criticism of "AI in general" in 2026, they're using it as a pointer to all of the social/environmental criticisms, rather than the technological capabilities.
We're all going to stay fixated on what exactly a "parrot" is, but the paper doesn't rest on the definition of tropical birds: it advances a specific and detailed hypothesis about how language models don't operate with communicative intent, and in fact the meaning humans obtain from LLM output is a cognitive pattern matching illusion. Those claims have not held up at all.
At this point, I think the authors are really counting on people not to read their paper.
How can you articulate a criticism without a repetition of your criticism becoming "inherently pejorative"?
You can't. That's just the way the news goes.
Essentially, what you are saying is that because some people somewhere frame a statement as pejorative, the statement itself is inherently pejorative. By that logic, every criticism ever articulated is inherently an insult.
Because they don't just parrot, they interpolate, which is why they have such varied abilities. You can't explain the range of behaviours they have with just parroting, and once you accept that, why shouldn't this qualify as some form of intelligence?
There’s a lot of confusion here on what parrots exactly are capable of doing and assumptions how smart they are. I also don’t quite see what you mean by behaviour in respect to LLM’s, possibly “agentic” tool usage? Because I’m fairly certain you can explain their behaviour as “stochastic parroting” combined with very strong program, ie. harness, to interact with other systems. And perhaps this all is “some kind of intelligence”, but then you have to be very careful with what you mean by intelligence. It becomes a terribly slippery surface described like this and you see, who knows where this gradient descent takes us if we are not careful.
So personally I would still rather undersell it. But parrot or not, it is still a terribly useful little bird indeed.
Really? Are you under the impression that parrots are able to synthesize their input and create entirely new, useful outputs which they have never heard before?
I think stochastic parroting is really a very accurate description of what they do (if underserving of the overall usefulness of LLMs). As long as you consider they are parroting from the whole of human intelligence. Its just that as they have gotten more sophisticated, the amount of gates, guardrails, and tertiary tools add variety. Trace any LLM hallucination back to provenance and you begin to see how the stochastic parrot works.
There's plenty of beautiful math there, but the relationship to what our neocortex does is pretty distant. Individual biological neurons can do fairly complicated things, including compute 10-bit parity functions (you would normally need a 3-layer MLP with a bunch of digital neurons to do this). And they don't seem to use backpropagation for learning.
Could you provide more detail? My understanding is that the neocortex is predominantly focused on forwards simulation, which seems distinct to how transformers operate.
That's fundamental to how anything that compresses/understands the world has to work, in the Kolmogeravian sense. That's why people denigrate LLMs as being just "next token predictors" - they're not wrong, but they're missing the point.
Because to do that kind of prediction out in the world you have to build up an accurate model of reality - a model that includes yourself! Which is why we and LLMs are self aware.
For the "how", it's been known for some time that LLMs operate on a Reimannian manifold - the semantic manifold - and that's a good place to start if you want to learn how they actually work; how a Reimannian manifold (plus some extra structure on top) can represent natural language in a form you can do work with is the part I find particularly beautiful. At a high level, the neocortex and LLMs appear to compute on the manifold in basically the same way - though a lot of the details are different; both are more sophisticated in some areas and less in others.
I'm afraid the precise connection you're making isn't totally obvious to me.
As far as prediction - I mean sure the cortex and LLMs do prediction, but then so can RNNs or diffusion models or any other generative model. Really any ML architecture is learning to compress its environment in pursuit of modelling. More broadly, the predictive brain model would suggest that all of the brain, not just the neocortex, is dedicated to prediction. What would you say makes LLMs similar to the neocortex, rather than the basal ganglia or Broca's area?
Similarly, if you agree with the Manifold Hypothesis, then all machine learning models operate on manifolds. I agree it's an exciting thought, but then I don't know what would distinguish an LLM from a VAE or SVM in terms of operating over a low-dimensional manifold embedded in high dimensional spaces - maybe just scale?
> More broadly, the predictive brain model would suggest that all of the brain, not just the neocortex, is dedicated to prediction. What would you say makes LLMs similar to the neocortex, rather than the basal ganglia or Broca's area?
The whole brain is most definitely not dedicated to prediction, and I don't think "prediction" is a very useful model anyways. You could say the hippocampus is for "prediction" if you really squint, but that's underselling what it's doing. And the basal ganglia operates off of prediction error, but it's more about regulating short timescale feedback loop learning than learning itself - speaking somewhat loosely because it's a particularly ancient structure in the brain and things get muddied.
LLMs obviously don't have functional equivalents to either of those - sure they can remember facts, but they can't intake new facts the way we can, the hippocampus is set up completely differently. But higher reasoning is humans is the neocortex, and that LLMs can do, and there's even structural parallels - LLMs and the neocortex are both implemented in layers, and you can even (with a shit ton of analysis) watch how LLMs walk the geometry across layers. The neocortex needs a lot fewer layers than LLMs, but that's because it's not pure feedforward across layers; the neocortex already does what the "looped LLM" people are trying to do.
> Similarly, if you agree with the Manifold Hypothesis, then all machine learning models operate on manifolds. I agree it's an exciting thought, but then I don't know what would distinguish an LLM from a VAE or SVM in terms of operating over a low-dimensional manifold embedded in high dimensional spaces - maybe just scale?
I view things more mathematically, and in math and physics it frequently turns out that there's really only one solution to a problem, or correct way to model something - and then everything else is just isomorphisms (which may be interesting objects in their own right!).
My interpretation/guess (informed by research, but with a lot of still unanswered questions) is that the manifold is fundamentally the structure that emerges from natural language: natural language has a computational model, just like Church/Turing machines have a computational model. Reimannian manifold is also underselling it a bit, it's also Finsler, and RoPE adds additional structure to that manifold (and when we get to RoPE it wouldn't surprise me if the neocortex and LLMs do diverge significantly in how they compute, but does the structure change? dunno yet).
Basically, I'm fairly confident in what I say about machine learning models that understand natural language; for non-NL machine learning, maybe the same structure emerges with enough complexity, but I haven't really pondered that.
Thank you for expanding! I come from more of an ML background so still learning a lot on these topics:
Agreed that the neocortex uses fewer layers because of looping - I also suspect it's partly because neurons are more complex than the neurons used in ANNs, so in principle they should be capable of more sophisticated computation in a single forward pass (especially considering that handling multiple neurotransmitters could mean superposed functions).
The point about there being one correct way to model something does seem backed up by the Platonic Representation hypothesis (https://arxiv.org/pdf/2405.07987). I've even seem some work that shows you can find a bijective map between the latent spaces of different transformers (https://arxiv.org/pdf/2505.12540).
Your point about Finsler spaces is fascinating - I hadn't come across the term before. It'd be interesting to see if there's work that specifically allows latents to exhibit that kind of directional metric behaviour, and whether that improves generalisation or something.
On the entire brain being dedicated to prediction, that's more of a high-level comment. I was thinking of work like Andy Clark on predictive processing, which suggests that even regions of the brain which we think of as receptive (eg. the visual cortex) may be implementing a generative/predictive model which is corrected by sensory input.
I'm not sure if the complexity is at the neuron level. It's clearly possible; we see some pretty complex behavior from single celled organisms - Stentors - and there were the experiments (starting in the 70s, I believe) that conditioned flatworms, ground them up and fed them to new flatworms and showed that the new flatworms had the same conditioning. Those experiments were replicated but never explained, but they hint at something like maybe memories being encoded in RNA.
It'd be wild if our brains used a mechanism like that somewhere, but I doubt it's in the neocortex because those would be slow processes compared to what the neocortex has to do.
I think the sophistication in the neocortex just comes from it using something more sophisticated with transformers. Most of the looped LLM research I've scanned through, they seemed to be training the model to "know" when it should loop more and think harder, and I don't think that approach will ever work - you're solving for paths on a manifold, so the looping needs to happen at the abstraction layer below the manifold, looping more if it's still converging. And architectures with more recurrence are just harder for us humans to reason about and comparatively easier for dumb evolution to find clever solutions, so my money is that that's where the neocortex has LLMs beat.
And I'd say the manifold hypothesis is quite a bit more than a hypothesis at this point. There's been work on production LLMs to use manifold analysis to optimize them - there was a paper in Nature on it back in January. Baby stuff compared to what could be done, but the geometric approach keeps popping up in mechanistic interpretability and seems to be showing the most insights.
The math is based upon theories of how the brain work, but even if those theories are right, this math is a great simplification and subset of what organic brains do.
This is a clear application of motte and Bailey. Motte: LLMs are stochastic parrots and don’t understand the text. They frequently hallucinate and are unreliable.
it annoys me how eager people are to hurl the word stochastic as pejorative. Statistics are a great tool for gleaning information from stochastic processes; statistics don't contribute randomness. Random sampling is necessary in order not to bias a sample, it's not used to contribute randomness to the sample but to preserve/measure the underlying distribution. (not meant to imply that training is random sampling)
It's a pejorative only because determinism is what makes computers useful in the first place. You get a consistent result, every single time, unlike if you have a human in the loop. Because LLMs are stochastic, they have removed the thing that makes computers useful to us, thus it's a pejorative.
What do you mean by determinism here? That you ask the computer 2+2 and it gives 4 as you expected or that if you ask the computer 2+2 and it hallucinates 5, you want it to always hallucinate 5?
Which one it doesn't do for you? Does it sometimes answer 4, sometimes 5?
There are definitely models that will always give 100% of the time the exact same answer, bit-for-bit, given the same input and seed. There are generative image models you can run locally doing just that. But you can also run some the SOTA chinese LLMs at "temperature 0" and, given the same input, they'll always give you the exact same output.
Because it's just a machine doing computation.
In the beginning of LLMs some "engineers" have tried to hand-wave non-sensical explanation as to why LLMs couldn't possibly be deterministic but: the open-weights models that can be run in a 100% deterministic way are way more powerful than the SOTA models of back then, so those explanation were pure rubbish bollocks.
Now of course if you run a complex chain of events, with LLMs doing calls to other LLMs, where some of them go fetch infos on unreliable networks, with infos that may have changed, then, logically, you won't always get the same answer.
Anyone who has spent time with parrots would realise that they can understand the meaning of speech without knowing what the words mean. Then somehow the meaning of word parts, and then you will find them making new words out of other words. Very clever indeed.
So stochastic parrots could indeed be a good description of LLMs. But I think that she meant it as a diminishing term (against the technology) which is pointless. Probably more of a reaction against SV tech bros than more nuanced interpretations.
This all sounds like a lot of backpedaling and “well actually” kind of stuff.
“Stochastic parrot got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way but that’s not how I intended it”.
Yeah that’s because it was chosen to be an insulting phrase.. Parroting is only ever used as a pejorative phrase. But sure, everyone else mindlessly parroting this line is the problem here.
This paper was always lousy, but it has really not aged well. We are living in a world when where an LLM has solved an Erdos problem. In a world where LLMs produce novel results that rival human thinking any conceptual reduction of an LLM is going to start inviting some unpleasant comparisons with human thinking.
Yes, and I don't understand how people like this paper authors mostly disregard all these achievements. It is obvious now that our common definition of "understanding" either is flawed, or at least needs redefining and precisioning.
What I look forward to after research like https://arxiv.org/abs/2603.02491, which demonstrate the necessity of world-modeling capability to achieve satisfactory performance on certain goals, is a refractor the SoTA test suites to demonstrate how much world-modeling is necessary in various task distributions.
There have been a few years now of arguments about the level to which transformers do or do not have a world model (v.s. being purely stochastic parrots like early pre-trained LLMs) and now we have some tools to actually make quantifiable determinations.
I think it's a good distinction to make between having and being, which seems to be what the whole "stochastic parrots" bit was intended to make all along.
It doesn't make sense to say a model is in possession of its self. That's exactly the sort of poetic anthropomorphization that Bender was criticising here, and a good reason to not refer to an LLM as "an AI".
Doesn't really matter that much what they're called as long as they're useful, and LLMs (particularly when harnessed) are already ridiculously useful. But it also begs the question: are stochastic parrots useful?
Yes, they are. Likely due to a deep relationship between math and physics, statistical modelling of complex natural phenomena has repeatedly been shown to be the most effective approach. This is true of LLMs, but also of many stochastic (and other) systems.
Useful for regurgitative pattern generation I can get. The core definition though is it doesn't understand what it's generating. Meanwhile I'm seeing LLMs constantly perform tasks which I'd say requires understanding, like reading the manual for a tool it's never encountered before and then going on to effectively use that tool. That's 2 different kinds of useful.
I'm of the general belief that something can be unaware and not have any understanding (in the subjective experience of consciousness), while also appearing to do so, and be useful.
That's what Bender doesn't get. Some folks tried techniques different from her favorite techniques and made unbelievably fast progress across a wide range of previously insoluble problems (regardless of whether they satisfy properties Bender believes are required for intelligent systems).
I think it's safe to say that none of the main LLMs have some sort of self-awareness as we think of it in humans, but I also expect that more sophisticated systems in the future could. If I had to guess, they would have significantly more activity going on in the network- not just individual end-to-end top-down forward graph, along with cycles instead of trees, and the neurons themselves would be sigifnificantly more capable (effectively little state machines that run functions on input that passes through). I guess also you'd want to have some sort of rules-based (but statistically trained) execution component managing everything.
I'm sorry but I do tend to feel like this muddies up the discussion on "what this technology really is".
I think "artificial" is actually a pretty good term to describe the output of the models. That output does appear to resemble at least some definition of the word "intelligence" - there is some ability there to do cognition over information that's been provided to them in-context.
What is it to understand, then? If they can work in complex domains and produce coherent output, it would seem to necessitate at least some definition of "understanding" of the corpus, even if that understanding is unlike how a human's brain would understand it.
What else should we call them then? They model language and information in ways that allow them to manipulate it on the fly. They do so 'unnaturally' from a human's point of reference.
I legitimately can't come up with a better term than 'artifical intelligence' -- not to be confused with artificial consciousness, which I don't think exists (yet).
Agreed, it's a problematic term that conflates theoretical research with better search results.
Deep learning and machine learning (ML) are both unromanticised (un-hyped) terms.
In some cases, it's just tooling with a better interface. As we have done with other complex computing systems (e.g. Deep Blue, Watson), we might just end up naming it a computing system for querying, e.g.
We would like a neutral term for such a system, and in this sense it's better to call it "A.I." than to make a verb from google or bing or other corporate name.
But in refined cases, such tooling may offer a credible (as in "believable") human experience, like Weizenbaum's Eliza (https://en.wikipedia.org/wiki/ELIZA). Some users of Eliza fully believed that Eliza listened and understood at a profound human level. Eliza was a simple computer programme.
Computing systems are not humans. They have no accountability in real life. People may be so comfortable with the user-experience that they cannot distinguish it from interacting with another human. That doesn't make the computer system alive and accountable. And if it's run by a corporation, it will almost certainly make big promises while energetically seeking to avoid accountability. ^_^
"Virtual intelligence" is better. Transformer ANNs are dramatically dumber than cockroaches and it doesn't make sense to describe such a system as being artificially intelligent, for the same reason it doesn't make sense to describe Half-Life: Alyx as an "artificial reality." An artificial reality implies some sort of scientific fidelity to actual reality. A virtual reality just has to be temporarily convincing. Likewise transformer LLMs have essentially zero actual intelligence - e.g. SOTA "reasoning" models still seem much worse at small-integer quantitative reasoning than almost all vertebrates. But LLMs have an enormous amount of formal subject matter knowledge and inexhaustible stamina at solving tedious O(n) problems. So for many purposes they are an adequate virtual intelligence. At least temporarily.
My source is "none of us have ever seen a robot that can navigate unfamiliar 3D spaces as well as a cockroach." If transformers were capable of the job we would have seen a smart robot by now. But all of our robots are truly mindless compared to the simplest insects.
I will change my mind if someone demonstrates such a robot. Absent this demonstration, cockroach-level AI is still an unsolved problem. Given how ignorant and arrogant and wealthy AI researchers are, it will remain unsolved. I don't think anyone alive today will live to see a robot smarter than an ant.
If you buy the argument “If A can do X but B can’t, then B is less intelligent than A”, then you have just as strong of an argument that LLMs are more intelligent than cockroaches. When’s the last time you saw a cockroach solve an Erdös problem?
Large language model is the term for what most people call "artificial intelligence". Bender's point is that labeling everything as artificial intelligence makes it more difficult to get funded or to regulate the technology. It's like walking into a car and being enamored by all the technology and saying "It's all computer". Yes, it's computers but that is not an accurate description of the many technologies inside that car.
Five years on, which term do we see as less accurate to describe LLMs? Artificial Intelligence or Stochastic Parrot? I guess it's still an open debate.
There seems to be some confusion between "we can" and "we should" in your comment. Bender (and others) are not discussing the capabilities, but rather (a) the fundamental mechanism(s) (b) the advisability and desirability of deploying systems that use these mechanisms.
There's no statement one way or another about should in my comment; and, for what it's worth, my ideal would be an immediate global pause in AI research and development.
But the different terms imply different mental models of what LLMs are and can do. If you take two people, one who thinks of them as "artificial intelligence" and one as "stochastic parrots" (with all the implicit context and connotations of the individual words composing them), what mental model would have led to better predictions of LLMs' future circa 2020?
The "stochastic parrots" phrase is very dangerous in that frame. People read far more into what capabilities it implies are (im)possible than the narrow technical description the authors originally argued for. If all they are is spicy autocomplete or pastiche plagiarizers, there's nothing serious to worry about. And when an opposition gets stuck in a trough that mindlessly dismisses their future capabilities out of hand because of a bad mental model, it renders them ineffective at preventing the worst outcomes.
>Which frame inspires a more productive research program?
This question depends on how you define research productivity. There is close to two hundred AI papers published every weekday. Most of them are about GenAI. Most don't seem to be all thay good. The progress in actual model improvement had mostly stalled. If you interact with the latest "raw" models they display all of the issues we've seen in GPT-3.5, just at a smaller rate. The "amazing gamechanger breakthroughs" I read about on social media every week do not seem to lead anywhere. It's all kind of boring, really.
The new "hotness" in AI is clearly building more and more elaborate harnesses. This is not at all the direction AI boosters have predicted couple years ago.
Personally, I think the "stochastic parrot" mental model is far more useful for science, because it primes people for proper testing, skepticism and researching alternatives. If you want useful AI, you want people working on it being skeptical, not credulous.
Statistical models have repeatedly shown themselves to be the most productive research method for working with complex human-based systems (and in the larger study of natural phenomena). It remains unclear whether there is any short term path for symbolic methods to catch up and exceed the capabilities of current/near-future statistical systems.
To me the real question begins only once we have a clear example of a non-trivial scientific discovery that is implicit (IE, not an obvious outcome of reading the literature and talking to the experts) and experimentally verifiable. Once that happens- especially if it is a reproducible process (IE, more discoveries) and it's significant (IE, impacts human life and mind in some profound way)- then the onus very much lies on Bender and her coauthors to explain whether we need more than a sufficiently advanced stochastic parrot.
> Which has better predicted the trajectory of capabilities over the past five years?
By that standard, parrots, and it's not even close. The framing of intelligence led to an enormous number of predictions that simply haven't been realised: an end to all white collar work, UBI, a total revolution in society, a literal robot god.
People are so desperate to view 'stochastic parrots' as dismissive that they misread the original argument while quickly ignoring all the failed predictions about how AI was going to overturn, save, and destroy everything.
The latter is definitely more colorful, and reflects a parrot's tendency to glom on to patterns. "Not X, but Y" being one of the more infamous ones.
Once in frustration I called a certain frontier model "Sam Altman's Tin Bird" to another agent with memory, and ever since then that other agent refers to ChatGPT as "the tin bird". Definitely a RAG artifact more than an attractor in that case, but I found it amusing.
This is a false dichotomy. Artificial Intelligence is more of a marketing term type of Hi-Fi or High Definition, ie. being a “suitcase word”[1], ie. it packs various different meanings and phenomena together to the point that without explication one cannot know what we are even talking about. Content recommendation system and LLM are completely different things.
What professor Bender is trying to explain here is that they were trying to describe how the LLM’s actually operate, to which point stochastic parrots is a fairly decent term. It is only disparaging if you know absolutely nothing how LLM’s work or you have some strange affixation to chatbots and believing they are far more capable than they actually are.
I think "(intelligent) language understander" is an apt term. It contains within it the fact that these models are mainly trained on text, and "understand" it beyond a simple token-by-token level (i.e. their latent space maps to more and more complex concepts).
It also separates them from "world understanders" since any understanding they might have about the world comes from text (or images if we include multimodal models). They do not gather experience, memories or other "qualia" that many people (me included) would probably include in a definition of human experience/intelligence.
(fwiw i think artificial intelligence is a good, broad term, but it is both too broad to describe the current sota, and too loaded nowadays to be using in nuanced discussions)
Sure, but that's the nature of language (which is also why i put "understand" in quotation marks. I usually follow it up with "whatever that means" lol) . I think in this case, it carries with it implicit properties (abstract semantic representation) that i think models possess, which a decoder doesn't.
Naming things using qualifiers like "large" has never aged well when transistors were involved.
For example, consider the term "short wave" radio which refers to wavelengths of at least 10 meters. Today's mobile communications use wavelengths 100x - 10,000x shorter.
Its less of open debate would say, and although superposition [1] is interesting, as a way to explain power of some effects, it is clear they are right now closer to Stochastic Parrots than AGI.
Why do I say that? Because you can trivially beat most guardrails, simply by encoding your prompt in base64 for example. :-) Just word matching...no real understanding.
Ask it how to prevent Spotify from automatically playing every time you get in your car. The answer will involve a bunch of Android settings that don't actually exist, cobbled together from a bunch of bad advice in online forums. Explain to it how it's wrong. Then clear your cache and ask it the same question again from scratch, and get the same garbage. Repeat until it's clear that it doesn't understand anything.
OK, so I asked ChatGPT "How do I prevent Spotify from automatically playing every time I get in my car?"
The answer looks entirely reasonable. I can't check the Android settings, but I was able to confirm that all of the suggested iPhone and Spotify and settings exist. Most of them I had already configured that way, some I didn't know about before.
Nearly all (99%+) people who use this phrase are anti-AI and just looking to show off how much they dislike AI and how clever they can be in insulting it.
So it's a great phrase because in just about every case I can ignore what someone says afterwards.
At least "glorified autocomplete" is technically accurate, even if vastly underestimating the capability of LLMs. It's just trying to make something very impressive sound trivial.
From an external standpoint, talking to another human, it's like the other human says one word and then says the next word. That's just how language works. Humans look like "glorified autocomplete" from this perspective.
I mean, looking at the time evolution of the state of the universe, one could say that all of physics and creation is "glorified autocomplete" to posit a next state of the universe given current and past state.
I dunno, man, I looked at that text and I see one word after another.
Obviously language and the connection to human thought is more subtle than this; I think we all have a rich inner life. Just from an external perspective we can't observe it; all we can see is the token/phoneme stream. I'm just saying that it's a mistake to try to criticize LLMs on this basis because it's hard to see how the same criticism would not apply to any system (like humans) that generate language.
If you want to see words form a shape I could point you towards concrete poetry, but I guess there is no point. Joyce wrote Finnegan’s Wake for 17 years and although superficially it seems complete gibberish, trodding through it you find meaning to words that are in no dictionary, sentence structures alien to English, etc. but still you are able to understand it, and perhaps some way the mind that produced it. So I disagree with you, we can observe each other’s inner life. It is always unexpected, strange, exciting, but always rooted to our shared experience or what it like being a very big and confused ape.
LLM’s are usually unexpected only when they malfunction and sprout same letter again and again etc - hardly a literary masterpiece. They make very easily recognisable patterns that we can use as helpful tools, but in the end they are devoid of any meaning apart from what we give them. Of course one could say same about art and all language, but I think there still is the fact that we apes somehow recognise each other. And besides, we do know the internal functions that drive the parroting. It is admittedly bit tricky, but in no way as magical as people purport it to be.
Oh, now I see where we have an actual difference of opinion. I don't think you can deny that even Finnegan's wake proceeds one token at a time; your interpretation of it may require more context or out-of-order interpretation, but that's just as true when observing text in German or Japanese, which have word ordering constraints that are alien to English speakers. How it was written is irrelevant; all we can observe is how it was presented. Of course we can observe each other's inner life, but we do so one token at a time, even if the process of producing each token is done (internally or actively) via a backtracking or zeitgeist approach.
You seem to believe, on a more fundamental level, that LLMs are simply not capable of producing text that has deeper connections to itself or represents abstract thoughts. In my opinion, 99% of text written by humans does not show this, just as 99% of text produced by LLMs does not show this, but both have the capability, and I don't believe that LLMs are constrained in such a way that they can never do this.
I am no linguist, but I believe this is referred to as surface structure and deep structure. What you describe is a line of text that is grammatically somewhat adequate to pass as readable and you treat all text the same. when we read the text, we decipher meaning out of according to word references and syntax - just like with Python or C++. However, if this were solely the case, we could not read Finnegan’s Wake. Probably vast majority of modern poetry would be unread by anyone, as would be pretty much all major works of philosophy Kant onwards. Deep structure is what according to Chomsky et al. gives meaning to the language, ie. somewhat logical structure behind the mere words. English word strict has order, need it but actually not does. You skibidi rizz swag grok brah also, barely. We use language in the extended meaning of the word to create a model of the world and somehow the past riverrun skibidi transmits that model to others. This is what I believe Bender also tries to say in their paper.
Now, you could claim that LLM’s have this deep structure, create models of the world and are basically just like us, and certainly many here are adament that this is the case, being aghast how someone can “insult” LLM’s by calling them parrots. However, there really is not much proof to back up this belief. Usually LLM’s seem to copy existing surface structure from whatever source, and when it deviates from these patterns, it usually becomes incomprehensible. There’s much hoopla about LLM’s solving hard maths, but it seems even there they are mostly generalising from vast amounts of training data, rather than actually reasoning: https://arxiv.org/pdf/2410.05229
What I'm saying is that we cannot observe deep structure. All we can observe is the surface structure. If we have access to the author or speaker we can ask clarifying questions, etc., but even that just produces more examples of surface structure.
But even here, in the Chomsky sense, LLMs clearly exhibit deep structure because they can write at length in an internally consistent manner. Importantly, early generations, GPT-2 and even GPT-3, did not definitively have this property; roughly, an object that was green at the beginning of a paragraph might not still be green at the end of the paragraph. This was strong evidence for lack of a world model.
Current LLMs do not show this behavior. We cannot prove that LLMs have a world model, in fact, their architecture seems to rule it out, but looking at it from a linguistic standpoint, they produce language in a manner as if to reflect a world view. That is, we cannot easily falsify the statement "LLMs somehow represent a world model"; and current examples of "disproving" their world view are so convoluted that even humans do not appear to (observationally) have a world view either.
I'm not making a claim as to LLMs having genuine deep structure or consciousness or anything like that. I'm claiming that we can't rule out current or future capabilities or make structural assumptions. Yes, they generalize from their training data, but unless you can make very specific claims about the kinds of things that they _cannot_ do, I can't take this statement as particularly compelling.
I disagree on your definition of deep structure. We can indeed deep structure, that is my whole point, without it Finnegan’s Wake would be impossible to read. This really is something LLM’s cannot do, at least for now. And if you look at the paper I linked, if they had a world model they would not get confused on mathematics when we change the values of variables. So there seems to be plenty of things LLM’s cannot do.
And basically the burden of proof is on your side. As often is the case, discussions on AI start to resemble discussions centred on religion and belief in god. Your only line left here is “yes but you cannot prove LLM’s are conscious” - which I can’t, since proving that Unicorns do not exist is pretty senseless activity.
So basically I’d say LLM’s are stochastic parrots (or whatever term you want to use) until we can very definitely prove that they indeed think and have some sort of consciousness.
afaik before the final sampling, every "next" token has a probability, so theoretically it could select the 10 most likely tokens (based on some kind of sampling algorithm), but you'd end up with exponentially many output-sequences, so nobody does that.
I think the point the poster above was making is that it doesn't predict a phrase or anything like that - just the single next token. So all 10 or 1000 or whatever number of tokens you want are each individually candidates for the single next token, not a sequence of 10 or 100 next tokens. If you wanted to create multiple possible seuqneces, you'd then feed each of the 10 tokens to the network in the initial state, and extract the next token (or 10 next tokens) from that one, than revert back and feed another single one of the 10 tokens, etc.
The term is not very useful since most humans are stochastic parrots... At least most of the time.
Not suggesting that I don't say stuff on autopilot sometimes but for many people, it's their only mode of operation. They never actually think about anything from first principles. Their whole approach to language is just chaining catchphrases together. It's how a toddler thinks; it seems like many people never moved past that stage of development.
It sometimes feels same as with the models, especially in corporate:
- Lots of Haiku around, many mistakes unless process is very clear
- Some Sonnets, still do mistakes but can adapt
- Some Opus, able to improvise and think outside the box.
But even the Human Opus/Mythos are hilariously wrong sometimes.
Conversely, that the most prominent proponents of LLMs call them artificial intelligence and then treat them like slaves they're free to abuse ought to be horrifying.
The "parrot" part of "stochastic parrot" is quite an ambiguous choice. Taken literally, it's referencing an animal that's actually quite intelligent and capable of complex, novel tasks but has no way to connect those to human language. How I've always read this though is the more literary meaning of "parrot" as "a thing that repeats words with no context". Perhaps "stochastic photocopier" would be a clearer metaphor.
Humans are not stochastic parrots. You are 100% wrong about toddlers. This was clearly explained by St. Augustine 1500 years ago:
Did I not, then, as I grew out of infancy, come next to boyhood, or rather did it not come to me and succeed my infancy? My infancy did not go away (for where would it go?). It was simply no longer present; and I was no longer an infant who could not speak, but now a chattering boy. I remember this, and I have since observed how I learned to speak. My elders did not teach me words by rote, as they taught me my letters afterward. But I myself, when I was unable to communicate all I wished to say to whomever I wished by means of whimperings and grunts and various gestures of my limbs (which I used to reinforce my demands), I myself repeated the sounds already stored in my memory by the mind which thou, O my God, hadst given me. When they called some thing by name and pointed it out while they spoke, I saw it and realized that the thing they wished to indicate was called by the name they then uttered. And what they meant was made plain by the gestures of their bodies, by a kind of natural language, common to all nations, which expresses itself through changes of countenance, glances of the eye, gestures and intonations which indicate a disposition and attitude--either to seek or to possess, to reject or to avoid. So it was that by frequently hearing words, in different phrases, I gradually identified the objects which the words stood for and, having formed my mouth to repeat these signs, I was thereby able to express my will. Thus I exchanged with those about me the verbal signs by which we express our wishes and advanced deeper into the stormy fellowship of human life, depending all the while upon the authority of my parents and the behest of my elders.
Humans learn language opportunistically. Toddlers start with a powerful "superchimpanzee" understanding of the real world, and use that to learn words in order to satisfy their needs and desires. Statistical frequency is incidental to what words a toddler learns: what matters is the real-world context. Also note how important it is that infants instinctively understand nonverbal communication.
The most depressing thing about the 2020s AI summer is watching ignorant tech workers use the success of LLMs to launder their own ignorant misanthropy. Your views are many many centuries out of date.
Toddlers do not actually start with a highly advanced "superchimpanzee" mind. Instead, adults project their own mature logic and thinking onto a child’s simple, playful actions.
> "The basic topic of my address today concerns how much of cognition is in the head of the infant and how much in the mind of the theoretician. My general stance is that we are being treated to an interpretive flavor of infant behavior that is much too rich."
FWIW nothing in this comment refutes any claims made in the comment it replies to. It's probably not the worst thing in the world for humans to start being a little more humble about themselves and their capabilities. Anthropocentrism has been a fucking disaster.
This is a complete misunderstanding of how even idiots function in the real world. There is a lot of thinking that goes into living a human (or even animal) life that models are nowhere near ready to model yet. Even ignoring the physical interaction side, the way any human sets and achieves long term goals (such as getting and maintaining a job), interacting with the huge amount of systems present in day to day life, and learning new tools along the way for decades is far beyond the current abilities of these models - even if they handily beat 90-100% of humanity on some tasks normally considered much harder.
> Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
I think this metaphor is so strained as to not be useful. I think key here is that the authors say "without any reference to meaning", which is a heavily loaded term, that does definitely apply to parrots, but does not apply when you apply it to immense bodies of text.
Namely that language embeds meaning in language. A sentence being written by a human (as a starting point) is designed to have consistent meaning. While it is possible to write syntactically correct meaningless text, that is not what most of human language has done; the meaning cannot be removed from the text.
This I think is clarifying, from the same paragraph in the text:
> ... the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.
That's just facially incorrect. The training data is entirely about sharing thoughts with a listener. Else why is the text being written?
What I have been doing in many places—the octopus thought experiment, stochastic parrots, the phrase “synthetic text-extruding machines”—it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do
> Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances. O also observes that certain words tend to occur in similar contexts, and perhaps learns to generalize across lexical patterns by hypothesizing that they can be used somewhat interchangeably. Nonetheless, Ohas never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives.
This seems kind of obviously wrong at least in the context of coding agents. These models get trained on actual output of the previous version of the model doing its job, often "IRL" on a real computer/project. It's like O is in the conversation for years now and learning from his own interactions between A <-> O <-> B, where A is the human and B is the computer.
The idea O ontologically has never "observed" "these objects" or referents is philosophically strained. Have I observed the moon, or a finger pointing at the moon? Have I observed `sed` more than Fable?
> It argued that large language models (LLMs) generate text by statistically predicting likely sequences of words rather than understanding what they are saying—a process the authors captured with the metaphor of a “stochastic parrot,” a system that repeats patterns without comprehension.
I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
I think it’s pretty clear that they are repeating without “comprehension” - both mechanistically (as in there is no facility for comprehension in their formulation) and in the ways they fail. The standard rs in strawberry, should I walk or drive to the car wash, etc all play on the fact that they don’t have any real world model or thoughts against which they can judge their output, as do many of the jailbreaks which basically play on the fact that the model has memorized patterns.
There are people who argue semantics, that we can call the pattern matching that LLMs do “understanding”, or the moronic “how do we know that’s isn’t all we do” but for the normal use of comprehension, LLMs at a fundamental level don’t.
there is an entire genre of riddles based on this kind of misdirection that works on humans, such as "A plane crashed on the border or US and Canada. Where do they bury the survivors?"
As with humans, comprehension is not a binary property of the agent - it is a quality that can be present in some situations and absent in others. LLMs may emit correct outputs sometimes because they do comprehend the input, and emit incorrect outputs in other cases when they do not comprehend the input.
In order to show that LLMs can't comprehend, we'd have to show that there are no (or at least very few) situations in which they exhibit comprehension, not show that there are some situations in which they don't.
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
So this seems obvious to you, and yet to many others, it is equally obvious that LLMs can/could do the things they routinely do without any meaningful sense of "understanding".
I think it's a mistake to disentangle their abilities from understanding. Just swallow the pill that they have some form of understanding, even if it slightly differs from ours. I really don't see the problem.
I prefer to work the other way around. That is, accept that a lot of human speech (and text) is generated via similar mechanisms to the ones that drive LLMs, but note that there is another kind of behavior - reasoning - which seems to be distinct.
I think you need understanding to reason, but you don't need reasoning to understand. A child understands how to catch a ball without reasoning about forces, air resistance, gravity, etc.
I think LLMs understand without reasoning. They've built a large associative network of concepts (a kind of understanding), but we don't yet have a good handle on the process of reasoning using that network.
I don't think it is useful to say that a child "understands how to catch a ball", even though it is something many of us do say quite often.
The child knows how to catch the ball, without understanding. Later, the child learns both reason and physics, and can reason about ball catching in a different way.
I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.
> The child knows how to catch the ball, without understanding.
I disagree that decoupling "knowing" from "understanding" is well motivated or useful in any way.
> I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.
This is starting to seem like a Chinese Room disagreement, so I'll just say that what that thought experiment demonstrated is that there is no robust or meaningful way to distinguish semantics from syntactic processing given everything we know about reality, and that LLMs were a fantastic demonstration of this illusion, eg. that "semantics" are reducible to a relational network of syntactic elements.
But it shouldn't even be contentious like that. It's not a fundamental mystery how these things work. It is for the most part not a valid target for the kind of speculation you seem to want to do about it.
It's not like you can be agnostic, or measured about this. It's like someone explaining a car to you, saying, "look here is where you put the fuel, here is where it ignites, where the axels are turned..." And you, trying to be measured, are like "hm well yes of course that all is clearly important, but there is clearly just a bit of magic here somewhere, between all the different 'parts'."
The "magic here somewhere" in the car is in the design that reference aspects of animal anatomy (facial features, stance) and in the millions of dollars of advertising that prime the public with expectations about how they'll feel driving it, or how to see other people in the car. There's a direct connection there to packaging LLMs as chatbots, it gives them a recognizable shape and behavior that a lot of people interpret as consciousness and personality.
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
Is it possible you're making the following error described in the article?
> The fact that these systems are designed to mimic the way we use language makes it very easy for people to mistake them for other people.
Clearly you don't believe it's actually a person ("it's not right to think of LLMs as a box with a little homunculus inside replying to you"), but you do believe it's doing something a little bit magical. Is it possible because the interface is linguistic, and every other thing in your world that communicates with language is intelligent, that you're projecting something that just isn't there onto the situation?
I'm sorry if this line of questioning is a little invasive. But this is literally the "danger" the original paper talks about, and it seems an awful lot like you've fallen for it.
I'm not offended by the line of questioning! But I don't really follow it. I don't and IIUC Bender doesn't use "understanding" to refer to any kind of magical property. Understanding is the capability of using words as consistent handles to things in the exterior world which the language is describing. And this is something LLMs can clearly do. I just went to ChatGPT and asked this question, which is almost surely not in its training data:
> What would happen if I walked to the top of a skyscraper with a soda can full of Maraschino cherries and let them go?
And its answer (https://chatgpt.com/s/t_6a4bd9ffa5708191901bb6d43c89f43b) clearly demonstrates understanding. It knew that this is a dangerous thing I should not do in real life, and that my question is ambiguous about whether I intend to drop the can, and that this might be intended as a physics problem rather than a real life scenario.
> It knew that this is a dangerous thing I should not do in real life
From the ChatGPT response you linked, all I see for sure is some matches on the following patterns:
drop $thing from skyscraper --> bad behavior
drop $thing --> physics
can of $stuff --> contents in/out of can
Then there are some sentences of likely characters following those patterns. You don't need anything more than a basic cartoon-level understanding of how an LLM works to explain this output. I see no evidence of reasoning or understanding here, or any theory of "real life".
It also does an incredibly poor job of answering your question. It makes no attempt to explain what might actually happen. If it has been trained on the entire corpus of medical science, and it is indeed intelligent, then surely it can reference ballistics studies and give you a very detailed and thorough theory of what--exactly--injuries you might expect from a 12oz can being dropped from the height of a skyscraper. Calculating the terminal velocity and therefore the momentum of the can is trivial. Characterizing the physics of the impact on various parts of a human body is trivial. If it actually understood your request why didn't it just answer the question?
It's a rhetorical question. LLMs do not "understand". It is completely outside their capability. "Understanding" is something we impose upon their output (to loosely quote TFA). [edit] I think the most powerful evidence for a lack of any understanding whatsoever is all the stuff about the cherries being in or out of the can. Yes, cans contain things. That is not a profound observation, nor is it at all relevant to the question. If you drop an empty can off a skyscraper nothing meaningful will happen. And, no, probably dumping all the cherries out won't hurt anyone or cause a slipping hazard... It's also not particularly relevant to point out that dropping things off skyscrapers is bad behavior. But that's more forgivable from a CYA standpoint.
I believe you are projecting something that is not there onto a completely mindless stochastic process.
This is a facile point. Lisp expert systems transparently don't understand the meaning of any symbols they process, yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability. The fact that LLMs are less transparent than Lisp expert systems (and easier to program) is extremely bad evidence that they understand language. Especially given that AFAICT Opus does not properly understand concepts like "four."
> yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability
Where can I access such a Lisp expert system?
If I cannot because they don't exist: then they cannot do the same things an LLM can do. And of course one can assert anything and everything about what a non-existing thing could do.
After having used LLMs for some time now, I don't agree with the concept they are just token generators, unless you think that's all humans are too. The way we test in most schools is just picking the right token. We also give them unique problems that they never saw in their training, which is the nature of programming. I realize they are probabilistic token generator models, but I find it harder and harder to accept that somehow there isn't something more going on. I'm not sure whether they are intelligent or not, but for the most part token generation is how you get degrees too. The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
Here's the thing: most things people do does not involve tokens of any kind. It is, in fact, stuff that not easily describable. For example, it's trivial for a person to walk, but they cannot verbally describe what muscles they're activating in what order to make that happen.
Cognitive skills such as tool use and complex navigation predate language as well. That means there's a core of reasoning in humans that doesn't depend on "tokens" or "language" of any kind. Language is a tool for communication and forming complex human societies, but it's not cognition.
> The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
Well a parrot does perform complex reasoning on novel situations all the time. It just doesn't have the wiring to connect that to "tokenized" human language. I suspect LLMs have the opposite problem, where they exist in the domain of their "tokens" and have no way to connect these to truly novel situations that have no existing words to describe them.
I think this is the most measured take I've seen from Bender, and I think it summarizes her only compelling point well (technologies should be referred to specifically rather than generally as AI, and that referring to everything as AI is not useful and helps hype the technology in a way that benefits those selling it).
In her previous interviews, I've found her assertion that LLMs aren't useful and will never be good at anything totally uncompelling. Also laughed at this quote as she's been pretty harsh IMO on "the people who like the systems".
> it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.
> in part because Google fired two of the authors, Timnit Gebru
I remember being angry about this situation when I first saw it on social media, until I read the details: This person submitted a list of demands to her employer and said that if they weren’t met, she quit. Google wasn’t going to meet her demands so they considered it acceptance of her resignation. There has been a movement trying to debate whether it was a firing or resignation ever since.
The original paper they published gets recirculated every year or two as some landmark history of AI safety, but as other commenters have noted it wasn’t really a great paper nor was it groundbreaking at the time. If not for the controversy surrounding the resignation/firing (depending on your POV), I don’t think it would have been notable.
I think part of it is she had excellent PR skills and a dedicated fan base. I was at Google when she quit, working in ML, and hadn't heard of her until the story broke. I remember there were a large number of Memegen posts about it, but no one I spoke with knew about her, so I assumed it was brigading.
I think she's since since lost a lot of her allure, especially when she didn't change her mind when the facts about the AI water usage changed 1000x
What exactly are the facts about AI water usage? I have trouble separating hysteria from reality but most of what I see still claims water usage is enormous
I like this summary https://www.andymasley.com/writing/the-ai-water-issue-is-fak...
And this is a reply to her comment about water usage where it becomes clear she's not arguing in good faith. https://x.com/AndyMasley/status/1990498830131888173
The hysteria around water usage rests on people not knowing the scale of industrial civilization. First thing to do is compare any estimate of data center water usage with the water usage of almond farming. Or, if you want to focus on individual consumer choices, the water footprint of eating a hamburger.
> Or, if you want to focus on individual consumer choices, the water footprint of eating a hamburger.
To drive this point home: if every American ate exactly one less hamburger per year, it would entirely offset the annual water consumption of all US datacenters (including, therefore, the water footprint of AI).
> if every American ate exactly one less hamburger per year
I fear that you have unwittingly led us further into the inevitable Environmental Hamburger-Cost Continuum.
Wherein we will use EHCC units to inform the hamburger voters about a) the nobility of even the smallest hamburgerian sacrifice, and b) the gentle, loving hand of industrialisation.
> the annual water consumption of all US datacenters
How many hamburgers will it be if all the proposed/pending datacenters are completed and running?
I prefer to use the hospital analogy. Locally, the water concerns are a big deal but a lot of people in my community are riled up about the potential need for diesel backup generators because of the noise pollution. They are not wrong and it's good to consider, but they are at this point grasping for reasons that would not (and have not) been concerns for other large footprint projects like hospitals with similar infrastructure needs.
And you don't understand why what's tolerated of a hospital may not be tolerated of other kinds of buildings?
Here in Reno the anti-datacenter crowd has been extending that tolerance to the local casinos and golf courses (of both of which there are many more than we arguably need). If the choice is between running servers to keep the Internet going v. letting poor schmucks get swindled out of their money on the slots, I'm picking the servers over the slots any day.
Breaking it down this way is a great way to minimize the numbers so that it appears reasonable.
See? Middle-Eastern investors are growing alfalfa in the western desert using legal allotments of water! That is so much worse than what we’re doing! Go after them!
They can both be using an egregious amount of water for silly purposes.
The other part of the water debate is also the pollution different systems create. Many data centres went in with the promise of closed-loop systems but changed half-way through construction and couldn’t be stopped.
I think it’s more complicated than, “they’re wrong, it’s just hype.”
Putting things into perspective is not minimising the problem. We literally have to do this to prioritise where our efforts can be useful.
Your argument makes sense if ai datacenters were using something close to like alfafa farming but the difference between them is soo massive it does not make sense.
Reducing pollution is a much better problem to fight for
I am guessing (not asserting) that there is a sort of cap on water used for agriculture. It's possible we've already reached it. (?)
So, on the matter of scale: there likely isn't a cap on water use of these datacenters. Both the heat emission and usage levels for these systems will likely go up unless there is a fundamental technical breakthrough.
On the matter of utility: As a sibling of GP mentioned, the utility of food is clear.
On the matter of polution: I am not remotely read on waste water and contamination due to industrial agriculture. Is this also something where the judgmental scale is tipped in favor of food production vs cooling systems?
what is the utility of alfaalfa? Because as it stands it's essentially just a way to export enormous amounts of water from a place where there's little (California) to a place where there's none (the Arabian peninsula).
This is bullshit to be charitable.
Your alfalfa is a red herring. Agriculture encompasses more than a specific crop that we can always legislate against production for export. The data centers will not go away and the utility of the "AI" they are to enable is as of today dubious.
> This is bullshit to be charitable.
What is bullshit is sockpuppets shilling for the very few who have bet the 'farm' on this and want return on their investment, the wellbeing of society be damned.
Everything we do has side effects. Some have more than others; in the case of water, even heavy users of AI have a relatively small AI water footprint compared other things they do.
There are ideological perspectives--another comment in this chain declares that anything that an LLM does has zero value, purely by virtue of being from an LLM. There's not much to discuss there: maybe I find alfalfa useless and LLMs provide a lot of value, maybe you do the opposite. Short of violence, either immediate or delegated, the best way to resolve this conflict is to incorporate costs for resources like water appropriately (accounting for externalities), and let market participants bid to determine who gets access to it.
Datacenter owners are highly likely to find this a reasonable process, because they believe that what they are investing in and operating will provide a great deal of value on the market. And other industries (like your alfalfa) are far more likely to throw a fit at this process.
Or if you think alfalfa farms have fundamental, deep-seated water rights that don't extend to data centers, then data center operators can just purchase water from alfalfa farmers, and everyone involved will eagerly make that trade.
some might call it "perspective"
Well the other thing is that agricultural uses are generally on a different scale and use different sources. And at least you can eat almonds and alfalfa. Probably more useful than asking an LLM to autocomplete your homework. Depending on your needs.
Even amongst municipal water usage, it seems hard to figure out how much DC usage factors in to other uses because often they’re not required, and therefore don’t, disclose their usage.
All I’m saying is that it’s not a closed case and not everyone complaining about DC water usage is a Russian propaganda bot.
Or even better: the footprint of doing something like farming corn for ethanol
Nice false dichotomy you got there there, might as well calculate the entire water usage for a a single GPU in the supply chain too. Something tells me one is extremely worse than the other when you account for all the water that's used in a single supply chain for high end electronics, but if you want to plop the measuring stick where ever along the whole pony show that makes you look better people will notice.
Also to compare growing food with the totally optional, not useful in the slightest, LLMs that somehow demand local populaces bend to their will for reasons that never seem to benefit them is just bonkers level of self-blinding when it comes to populations absolutely despising big tech, big tech leadership, and big tech practices.
This mania might finally cause the software industry to become a highly regulated with licenses similar to that of other engineering disciplines due to amount of optional destruction they have decided to unleash upon on the planet in such a short time frame.
let's compare ai water usage with something even more useless, water leaking out of pipes. the United States currently loses about 2 trillion gallons of water annually to leaking pipes. this is in comparison to around 230 billion gallons used by data centers.
we're literally dumping several times the amount of water used by data centers onto the ground for no benefit at all. oddly enough I haven't seen any protests about this despite how concerned everyone is about water usage.
It's because spilling water doesn't do anything: water evaporates, becomes cloud, rains down etc.
Meanwhile, datacenters sacrifice water by sending it to Mars in order to compute criminal deepfakes of innocent politicians.
This is a useful piece on that: https://andymasley.substack.com/p/the-ai-water-issue-is-fake
How do you know this doesn't suffer from Gell-Mann Amnesia? The first version had so many glaring errors that have been "corrected" (removed), and I don't have the energy to comb through this one.
I am highly skeptical of layperson debunking like this.
Andy has a good track record for writing about this. He shares plenty of credible citations - more so than most other people commenting in this space.
He also caught a major error in one of the most widely read books that helped kick off the whole data center water debate: https://blog.andymasley.com/p/empire-of-ai-is-wildly-mislead...
The thing that is off putting about how he uses rhetoric is that it feels like and-you deflection (tu quoque).
> Claim that a data center is using 1000x as much water as a city of 88,000 people, where it’s actually using about 0.22x as much water as the city, and only 3% of the municipal water system the city relies on. She’s off by a factor of 4500. This is the single largest error in any popular book that I’ve found on my own, and to my knowledge I’m the first person to notice it.
Yeah, she might be wrong. But a data center also using 1/5th of the water consumption of an 88k person city should still be what are debating. We also have a base rate fallacy, we don't know know how well or how poorly they are using water. Nearly all criticisms of AI data center criticisms boil (no pun intended) down to yeah, but what about almonds or rice or xyz. That isn't a healthy way to adjust how we talk about data center water consumption.
It is a classic debate tactic. Someone makes an argument, then buttresses that argument with a number. You attack the number, pretending that you didn't just correct the number, but also invalidate the original argument. We shouldn't be using these tactics to talk about the tragedy of the commons.
I worked for a hyperscaler, I poked around a bit about water usage both internally and externally and it wasn't good. There was little to no thought other than, "we can pay X to have water delivered, doesn't matter if it sourced responsibly." (to glibly paraphrase, company policy is to never write the honest part down)
Look at how hard Google fought to not have water data released in The Dalles Oregon for their DC there. Many DCs are supplied by water that is meant for humans, sourced from aquifers that took hundreds of thousands of years to fill, that are being depleted faster than they are filled, already.
I think AI is powerful tool, but we still can't give DC expansion a pass.
> I think AI is powerful tool, but we still can't give DC expansion a pass.
I really don't think we are. In the wider culture the idea that data centers use an absurd amount of water is baked in at this point.
It frustrates me because I think it distracts from energy usage, which is a much more real issue than water usage.
I'd love to see DCs scaled with renewables, they would throttle down and run slower at night. The abstraction that power is available at whatever rate you want it is an expensive myth at scale.
I defer plugging in my electric car until 10am so I know that my neighbors solar arrays are charging my car.
In Seattle, every watt saved is water saved since we are blessed with hydro.
> But a data center also using 1/5th of the water consumption of an 88k person city should still be what are debating.
Why, when that's such a minuscule amount of water in the grand scheme of things? Why focus our energy there? Why not spend that energy on the myriad many-orders-of-magnitude-worse offenders?
So I'm not sure if it's the first version of this doc, but I ran the first version of it that's on Wayback Machine[1] with the current version to see what the differences are.
Most of the changes seem to be because Masley found one counter-example (Newton County, Georgia) where AI datacenters do seem to be increasing water costs; the only deletions AFAICS is toning down language where Masley used to say "there are no examples" to "there is one counter-example". I don't see any other major corrections that have been removed.
Here's an annotated diff of the two texts: https://bafybeie7b3zs2gqifpvn7ee7y7326wcexwnsbhnur5coymu3m6w...
in my agi doomer opinion the water argument is akin to a concern that dropping a nuke may endanger certain rare species in the area
What is interesting about it for me is: why would someone working on AI ethics choose to work for Google at all?
Did she really think Google cares about ethics? Such positions seem purely performative, we all know that ethics go out the window first to make room for more profits.
[flagged]
I missed that thread when it came out, it's really a wild read. The difference between how people describe her vs how she's normally portrayed in the media is really startling
People generally get fed the stories they already believe to be true
So that's why "Brave minority woman unfairly fired by evil AI corporation" sells better than "Self-entitled minority woman is terrible bully to colleagues"
True but also... she wasn't a software engineer putting code in production nor a researcher working no the fundamentals of machine learning negotiating a raise.
She was part of the "Ethical Artificial Intelligence Team" of what was then, and still is now, one of the corporations World wide spending the largest amount of resources precisely on using AI commercially.
I'm saying the paper itself wasn't a bombshell or even that noteworthy. The reason it got PR and continues to come up was because the authors manufactured this self-inflicted drama around it, not because it was leaking secret revelations that harmed the company.
Never underestimate the power of a catchy title that resonates with the intuitions and preconceptions of people who will never read the paper.
Cf. https://machinelearning.apple.com/research/illusion-of-think...
Timnit got popular because she was part of Woke 1.
Proof here: https://xcancel.com/timnitGebru
TLDR: if you're speaking about tech in public, and you're white, she probably has published a couple of hateful posts about you.
This is an extremely unserious person.
Her demands included wanting to know the identities of anyone who wanted to comment on her paper, after she had a history of going after people publicly. That's enough right there, nobody should tolerate toxic behavior regardless of whether you agree with the politics.
Meanwhile, the paper has 2 points of criticism towards AI. 1 is a bunch of carbon consumption complaints assuming NVIDIA cards with coal-fired power, while a lot of effort at contemporary Google went towards getting TPUs running on green power. I suspect this was what people wanted to object to, a lot of effort went into those green power projects and she was just denying it. The complaint seems prophetic now but it was not true about Google then.
The other criticism was about which language the LLMs use, they average the input data of normal humans instead of talking the way the paper author thinks they should talk. The phrase "women doctors" is called out as problematic. I'm less inclined to think people objected strongly to this given the zeitgeist at the time, it was probably people who worked on the green energy projects and were pissed off that their contributions were ignored, but still, nobody elected her Queen of English, she can have her opinions but she's not a victim for not having them adopted by everyone.
One of the key points of that paper is that the body of written works is biased and those biases will be amplified by compressing that body into LLMs, with outlier data, with low coincidence rate, being suppressed as unreliable - data that is structurally dissimilar to the bulk is noise. Similarly to how PageRank suppressed nodes with low number of edges, probably contributing significantly to the homogenous corporate mall-internet we enjoy today.
I think this is completely misleading. If your employer asks to redact your paper because it ignores relevant research, naturally you want to know what research they are talking about and on what grounds, and also which researches reviewed the paper (probably assuming there was none, review process was used as an excuse to silence criticism, G has done it many times).
(BTW, quite bold to say input data from Reddit and 4Chan is how “normal” people speak. There is a lot of language in the training data of any model you really do not wish your application to use ever.)
Ugh. Ok if you're going to push one-sided propaganda I'll push the other side.
Google forced its researchers to retract an already submitted paper because it undermined its strategic and commercial story around large language models. The "we just accepted her resignation" is just a lie. Google made harsh demands with opaque reviewers that made vague objections, and then Jeff Dean moved very quickly to get rid of Gebru. Other Google researchers reported that they usually got to work through objections, Gebru got no such opportunity. Google showed that AI labs will not tolerate internal research that seriously criticizes technology central to its business.
Dean pulled a sweet Dungeon Master move in "accepting her resignation." She should have made them fire her, esp for ostensibly doing the job she was hired to do.
I think a better summary is that google gave someone the responsibility to ensure AI was developed and used ethically, and didn't give them the power to execute that responsibility.
Apparently google did not even give them the freedom to come to their own conclusions.
I just skimmed through the paper again and what it says seems to hold up well.
The paper is heritical in that it suggests slowing down, using carefully chosen less biased data, and understanding how it all works.
Basically challenging the AI bitter lesson.
I can see how that would meet with a lot of resistance from people who have gone all in on the bitter lesson.
I'm surprised that people still take Gebru seriously. She is a disgrace to the community because she always, I mean literally always, attacks her critics by motives. You think bias is a data problem? You're a bigot (See her dispute with LeCun). You disagree with my assessment on an ML model? You are white male oppressor (her attacking a Google's SVP). Oh, did I mention that she even said that some loss functions are more racists than others on X?
Gebru is not a researcher. She is a modern-age Trofim Lysenko, who politicizes everything and wields political correctness as a weapon to purge any dissent.
Pressuring an employee to add unethical behavior or specific religious practices to their job description is constructive termination.
I'd say what's under debate is whether uncritical LLM adoption is mainly unethical or mainly religious.
This is not why she was fired and wouldn't have been a plausible reason in 2020 when it happened.
>Timnit responded with an email requiring that a number of conditions be met in order for her to continue working at Google, including revealing the identities of every person who Megan and I had spoken to and consulted as part of the review of the paper and the exact feedback. Timnit wrote that if we didn’t meet these demands, she would leave Google and work on an end date. We accept and respect her decision to resign from Google.
This is Google's side of it; I think the following is a fair piece of primary-source journalism if you want to go deeper:
https://www.platformer.news/the-withering-email-that-got-an-...
I don't see why it's those are the only two options, nor why they are even mutually exclusive.
Here is what Jeff Dean said about the firing at the time: https://docs.google.com/document/d/1f2kYWDXwhzYnq8ebVtuk9CqQ...
She was probably repeating behaviors she learned in academia. These kinds of extremely toxic "don't just apologize for disagreeing with me but also give me the name of every person involved for collective punishment" is a classic move for academic tyrants throwing weight around. The understanding there is that they will then move to cut off every named person from power or access to academic resources. Google did the right thing by protecting their people.
I have watched it happen multiple times that someone from academia joins a research group at a large corporation and finds out to their chagrin that they can't just overtly bully colleagues as easily without tenure.
edit: And it looks like Mitchell immediately left an angry comment about being called a "linguist" despite the article never doing so. Starting to notice a pattern of confrontation, ego, and escalation among some of these people.
> resignation
I appeciate short letters like this that get straight to the point...
You can always count on management to tell it like it is /s
> With the octopus thought experiment, I initially had told the story in terms of a dolphin, because dolphins clearly are intelligent animals. My co-author on that paper, Alexander Koller, said it should be an octopus, because first of all, the environment that octopuses live in is much more distinct from where people live. It makes the metaphor more vivid, that the octopus is just feeling these pulses in the cable and has no way to look at what the people are looking at.
On a completely tangential sidenote, octopusses are actually very very intelligent: https://www.nhm.ac.uk/discover/octopuses-keep-surprising-us-...
It's such a tragedy that they're also extremely solitary animals and die shortly after reproducing the first (and only) time.
Almost all other particularly intelligent animals seem to be gregarious, and it's easy to conclude that a social lifestyle tends to select for more intelligence, a sophisticated theory of mind, and so on (I like to think that that's exactly what was responsible for a runaway intelligence explosion in humans). But in the case of cephalopods, there's something else that has been applying selection pressure towards exceptional intelligence.
I agree, which is why I think this species might be the start of something amazing: https://en.wikipedia.org/wiki/Larger_Pacific_striped_octopus
I've never heard of these before, that's fascinating, thank you!
Is it a new species though? Or just a living fossil that exhibits behaviors that were bread out of all other octopi species?
It's hard to imagine how high intelligence could be selected for without social component that provides the individuals a lot of pressure to outsmart each other.
Intelligence is plenty adaptive when you're just trying to outsmart the environment.
The notion that this is a "living fossil" is... pretty wild. It's genus octopus. Unless it's wildly misclassified (doubtful), it has a recent-ish common ancestor with most other common octopuses, which all have normal octopus behavior. You find yourself in need of extraordinary evidence.
> Intelligence is plenty adaptive when you're just trying to outsmart the environment.
I think there's very little evidence for that. Environment is just very slow when compared to outsmarting members of your own species that you tightly share the living space with. I don't think you appreciate how abnormal is high intelligence. How pathological conditions must have been to trigger development of something that bizarre and costly. This must have been run-away self-reinforcing feedback loop. We know of many such loops between predators and prey and none of them look strong and fast enough. Where we find intelligence, it's almost always tightly connected with social behaviors. It would be exceedingly strange if there was a true exception.
> The notion that this is a "living fossil" is... pretty wild. It's genus octopus.
I didn't mean that literally. What I meant was that the common ancestor of this octopus species and others might have been more like it, social, not like the rest. Maybe this species preserved of what normal octopus behavior used to be. If the solitary life became more evolutionarily favorable after the development of intelligence most descendant species could have switched.
> You find yourself in need of extraordinary evidence.
Yeah, maybe at some point we'll discover a huge collection of beaks in some under-ocean sediment and it will become obvious. Ocean floor is not really that explored.
> I think there's very little evidence for that.
The evidence is literally every other evolved form of intelligence. Including, despite your speculation, octopuses. Do recall "the environment" also includes other intelligent agents an animal needs to compete with. Does that help you understand? A runaway process might be required for human level intelligence but clearly not in general.
Basically all octopuses are solitary and die after breeding. LPSOs are nested in one group of octopuses. You're proposing that one group of octopuses developed social behavior and multiple breeding, then had a bunch of descendants who went back to exactly the normal octopus behavior. Hopefully it's obvious why this fails Occam's razor.
> > I think there's very little evidence for that. > The evidence is literally every other evolved form of intelligence.
Not really, even harsh environments don't really select for intelligence. Neither does predator-prey dynamics. There are very few significantly intelligent species, and except for octopuses they are all social.
> Do recall "the environment" also includes other intelligent agents an animal needs to compete with.
If you include members of its own species in "the environment" we don't have any thing to discuss, because I fully agree that "the environment" understood as anything external to the individual can trigger the development of intelligence. Because it's trivial to observe that it did.
My claim is more informative (if true), that for the development of intelligence the environment must include populations on individuals living in close proximity, social, interacting a lot, where the strongest evolutionary pressure on a given species is the species itself. That doesn't seem to be true for most octopus species today. As an exception to the rule, their intelligence is interesting. I personally believe it comes from the same source as every other intelligence. And as the living space for octopuses expanded, they became more solitary. Maybe there's something general about this too. That with sufficiently intelligent species (perhaps if the intelligence can't rise anymore due to physical limitations), the other individuals present such a high danger that the species must spread apart and become more solitary. Maybe humans reached this stage already and spread apart, but as they ran out of Earth, they were forced back into old, more social mode, just now on the planetary scale. There is evidence that humans in Europe lived in way smaller groups and used to be way more murderous and cannibalistic not that long ago.
> Basically all octopuses are solitary and die after breeding.
That fits. There's no point in extending the life beyond reproduction if your kids are gona kill you anyways.
> You're proposing that one group of octopuses developed social behavior and multiple breeding, then had a bunch of descendants who went back to exactly the normal octopus behavior.
I'm not proposing that at all. What I'm proposing that all ancestors of octopuses were social and could breed multiple times. However, because of the proximity they became their greatest evolutionary pressure for themselves as they got more intelligent. Eventually they spread out due to some environmental change and their social behaviors and multiple breeding atrophied due to danger from other members of theirs own species, that now could just be nearly completely avoided. And that's the case for nearly all descendant species except this special one for some local environmental reason. This species shows us that social octopuses are (and were) a possibility.
What you consider normal octopus behavior is as normal as eyes of a molerat.
That's a fun story, but it doesn't match the evidence at any point. Your fixation on intraspecific competition as the only significant driver of intelligence is leading you far astray.
The evidence is the lack of any solitary, highly intelligent species apart from octopuses.
In Stefan Wul's SF novel "Nyourk", octopi evolve to become the Earth's dominant species, which was quite prescient back in 1957, when almost nobody knew octopi to be intelligent. :)
It's also a bit of a chicken-and-egg problem: if they were raised by their parents like all other more intelligent animals, they wouldn't need to be as intelligent as they are in order to be able to relearn "octopus behaviour" without help from other members of the species.
Also, last time I checked, the environment where octopuses live is actually the exact same environment where dolphins live?
Well in a sense that monkeys and Great Condor inhabit the exact same environment.
That's a bit exaggerated - monkeys can't fly, but both dolphins and octopuses can swim. I'm aware octopuses prefer to stay at the bottom, while dolphins have to come to the surface regularly to breathe, but for me it's still the same environment...
Tidal basins and ocean bottoms are very different from the open ocean surface. Much like tree tops and 3 miles in the air are different. Both are up in the air, but still quite different, just like "ocean" but still different.
You know monkeys can swim? Do you know what happens to a monkey 2-6km under the surface? Monkeys can also climb trees, very high. Monkey still no condor, sadly.
The continued use of animal metaphors is doing them a great disservice. Esp as we learn more about animal cognition, on first look, it smacks of human exceptionalism that has littered the historic scientific consensus.
Now if they had said, "Imagine your average American ..." (/s)
> when OpenAI imposed ChatGPT on the world...
OpenAI offered ChatGPT to the world. A large, monied cross-section of the world had yet to throw its capital behind the Large Language Model technology that made the ChatBot possible. While it is fair to see AI development now as a global imposition, OpenAI did not have the agency as a 2022 startup to impose on the scale we see now.
I agree with a lot of her points but that word really is revealing of her thoughts about OpenAI.
> A large, monied cross-section of the world
I asked Mistral, and it guestimated that Altman, Thiel, Musk, and Hoffman had like $20.3B together when they founded it. Sound to me that the founding of OpenAI was exactly the point when the monied world threw its dollars behind AI.
$20.3B is far from the trillion+ investments that are causing enormous societal contention now.
I think "imposed" is a pretty fair word. LLMs already affect everyone's lives whether you use them or not and their footprint is still growing. It's like I don't have to buy a car, but cars are absolutely imposed on everyone anyway.
Bender's linked May 12, 2026 post "Frequently Unasked Questions", https://medium.com/@emilymenonbender/stochastic-parrots-freq... , was a better read.
Her language consistently defines LLMs in negative terms like “synthetic text extruder” but she claims she’s not trying to denigrate it. What’s missing for me are similar terms from her about how humans create sentences and thoughts. Judging by the state of the internet humans are quite capable of making shit up to argue their point (see latest Fox News apology). She talks about sycophantic AI but give me a car battery and some cables and I can train a sycophantic human (no I can’t but there are people who can). She’s pretty much a walking counter argument for her own claims.
If your field (NLP/computational lingustics) has evaporated the next best thing to do is hit the pundit circuit.
No you see, she says right there in here "Frequently Unasked Questions" (lol) that she is in fact not at all bitter about her field being effectively solved by someone else, thank you very much! Her preferred approach is in fact alive and well, more important than ever and still very very interesting. One wonders if her PhD students feel the same.
The authors of the paper make some good points (in it and elsewhere), but I've seen precisely nothing that suggests they aren't awful people who resort to playing the victim card, inciting cancel mobs and in general behaving like the worst stereotypes of "leftist academia" in response to any criticism.
I paid a bit of attention to this paper and the phrase 'stochastic parrots' when it came out and i thought this was worth saying and doing at that time. their suggestions about financial and environmental costs are worth studying, their concern about carefully evaluating datasets to feed to the model rather than feeding the entire internet is fully justified. so - to everyone saying this was a bad paper; if you have actually read the paper then please list a few criticisms. all i have seen is "oh this wasn't that good of a paper" or "can't believe how bad this paper was".
My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation.
It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!
Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.
not sure your criticism makes sense though - they did this pre chatgpt. they are talking about the language models of that time. they did not make predictions about the future.
They made a claim about language models in general, not just ones that had been released so far.
The point of the paper, in fact, is that language models are getting "too big", and another approach is needed to make progress, so they were certainly predicting things about later models.
With that said, they talked about "pure" language models, so it is fair to say that they didn't talk about, say, LLMs that are multimodal or that have tool use, which are advances that happened after their paper.
They did use RLHF at the time, at which point it is not a pure probabilistic representation of the training corpora. Bizarrely, RLHF never came up in the paper.
Statistical operation doesn’t preclude logical processing.
We’ve know that since 1943 when McCulloch-Pitts came up with the first “artificial neuron” definition. And since LLMs are a descendant technology — our assumption should be they’re reasoning in some internal learned logic.
This is what the evidence supports — eg, the “stochastic parrot” crowd never can explain transfer learning. Whereas for the internal reasoning crowd that is easy: removing your top level judgments from a theory still leaves you with useful terms for describing a new theory — eg, removing your judgments about “which animal is this?” but preserving the underlying structure for representing an image in your new judgments, “is this cancer?”
There’s 80 years of reason to think DNNs reason and zero support other than “sTaTs R mAgIc!” to support the stochastic parrot interpretation.
Ignorance isn’t argument.
Those costs have to be compared to the way things are currently done without AI.
They never are. Ever.
>They never are. Ever.
And even when they are: they sure seem to bet against Moore's Law or just the general tendency for things to get better/efficient over time.
It's frankly remarkable how capable the models have become that we can run locally now on a decent laptop.
The same thing happened with image generation. I've had arguments with people that image generators are killing the environment, but I can do it in 20-30 seconds on my GPU. No one bats an eyelash when I play 20-30 minutes or even hours of a video game on my GPU, but the images are burning down the planet.
It's slightly maddening.
The contention that there is no grounding because the training data is linguistic and thus can only reference a world model is disproven in "This sentence has five words"- there's real, grounded information about what "five" means within that sentence. While that's a trivial counterexample, I don't know that it's an obvious one (I didn't come up with it myself).
It's not a criticism of the paper itself, but multimodal models came shortly after and provide grounding that is more of the sort the paper is getting at, and it didn't seem like anybody updated on that at all. If multimodal models were still stochastic parrots by the original argument, humans would have to be as well; we don't have any way to ground anything beneath sense data and evolution can't have programmed some innate grounding into us because it didn't either. But (and maybe this is my own misperception) nobody threw in the towel at that point.
I confess I never read the original paper until now, opting to absorb by osmosis instead, and I was quite surprised that they don't really make a deeper case than that. After just a few paragraphs about how they can't be grounded because humans don't express their thoughts directly, it lurches into a page about how they can be biased by training. And they certainly can be, but that has little to say about their stochastic nature- humans are biased as a rule with no exception. (For the record, I only read the Stochastic Parrots section before this reply.)
It's not really a bad paper, but I don't see why it ever carried the esteem it did. Hating on it is like hating on Taylor Swift- she's fine, yes, but for her level of success, one is inclined to question every dumb lyric where others get a pass. (Apologies to Swift fans, substitute a successful artist you don't care for here.)
To add to dwa3592's comment, a sentence is not a self-contained idea. The sentence doesn't include what any of the words in it mean, nor what "this sentence" refers to. The exact same sentence can mean different things depending on the text that surrounds it.
Fair point, and on its own it would be surprising to learn what "five" means from that sentence. But you can extrapolate- across a billion sentences, there will be "the next sentence has five words"s and "this sentence are grammared wrong" and so on. It would not be at all impossible to ground a world model on pure text for that reason. And 'not impossible' is sufficient to invalidate the paper's argument.
Let me provide a less superficial response, then.
>If multimodal models were still stochastic parrots by the original argument, humans would have to be as well; we don't have any way to ground anything beneath sense data
Animals don't passively learn from their perceptions, don't have a separation between training and inference, and don't have a prompt-response execution model. Besides its fundamental biology, the grounding an animal brain has is that when it outputs a motor signal it receives some feedback as to the effects of a signal of that strength. A bird learns to fly because the grounding truth of aerodynamics and gravity consistently respond in a specific way to the flapping of its wings. It doesn't learn by passively replaying thousands of hours of somatosensory recordings of flights.
A multimodal model doesn't have the capacity to do much with a prompt. It has no head to turn to look at an image from a slightly different angle to attempt to gleam more information, doesn't have the capacity to interact with the real thing the image represents in any way, and even if it requests another angle and is given it, it lacks the capacity to learn that new information permanently. A multimodal model knows about images of pipes and facts about pipes, but doesn't know pipes; it doesn't have literally first-hand experience with them.
>evolution can't have programmed some innate grounding into us because it didn't either.
What do you mean? Of course genetics programs ground truths. For example, "forward is the way your face points when your neck is relaxed" and "if you can feel it, then it's part of your body".
>and even if it requests another angle and is given it, it lacks the capacity to learn that new information permanently.
I'd argue this isn't true today, but that the loop for incorporation is long (ie, the next training or finetuning run).
>A multimodal model knows about images of pipes and facts about pipes, but doesn't know pipes; it doesn't have literally first-hand experience with them.
Wouldn't this mean that any human who hasn't seen a pipe in person or interacted with it, similarly doesn't "know" a pipe? Most of us haven't interacted with the vast majority of "things" in the world, yet we're still able to build a model and abstractions for them such that we can reason about them, right?
A bird doesn't learn gravity or aerodynamics, it has no 'sense of physics'. It has sensory neural activity that a scientist can show is tied to these things, but you could, at least in theory, falsify the entire experience of the bird. There is nothing in a bird's brain that directly percieves reality. Colors and sounds and textures and so on are all false primitives that don't exist in nature without us, if that's clarifying. And because evolution acts through organisms, its only access to base reality is through their senses. After thousands of years of effort, we have some pretty good models of what reality actually is, but they're still not 'grounded' in the sense you describe.
>A bird doesn't learn gravity or aerodynamics, it has no 'sense of physics'.
That's not what I said. What I said was that it's physics that provides the ground truth.
>you could, at least in theory, falsify the entire experience of the bird
It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions.
>There is nothing in a bird's brain that directly percieves reality.
Yes, of course there is. Animal sensory organs do not produce false information, nor do they provide the brain an interpretation of what they perceive. The brain may fail to distinguish hallucinations from reality, but those are processes internal to the brain. What it gets from the body is raw physical measurements, and what it sends out is raw motor commands.
A multimodal model doesn't have the same direct access to reality, it just has collections of words and images. It has no capacity to determine the reality of a photograph of a sunset or a CGI render of a dragon. The word "real" is itself meaningless to it; they're both real in that they appear in its training corpus. It lacks the capacity to investigate these stimuli in any way, and can just learn to associate different stimuli in arbitrary ways that appear to make sense to us, but nothing else.
> nor do they provide the brain an interpretation of what they perceive
> What it gets from the body is raw physical measurements
No, sensory organs like eyes do a lot of processing ("interpreation"). They certainly don't send "raw physical measurements" to the brain.
And even then, it's just sampling. Much of what we "see" is a prediction, and there are plenty of optical illusions out there premised on that (plus VR techniques like foveated rendering that take advantage).
>It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions.
The criticism in the paper is of the architecture of LLMs, isn't it? The paper contends
"""Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. [...] an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot"""
They're saying that the model cannot learn anything about reality irrespective of training data. Your point is an interesting one, but I think it's distinct. To your point though, this is unfalsifiable from the perspective of the "bird". I can't prove that I'm not a dysfunctional cyborg with false perceptions, which makes me wonder if that's a meaningful distinction.
>Animal sensory organs do not produce false information,
I happen to be in possession of some of these and I think this needs a "usually, under ordinary conditions". (Nitpicky and not critical to my point, but I liked the beginning of this sentence too much to edit it out)
>nor do they provide the brain an interpretation of what they perceive
Whereas this I'd argue is not true at all. My cones interact with wave-particle photons at particular wavelengths. I can't even conceptualize wave-particle duality (though some humans can), but "red" and "blue" are the bread and butter of my visual consciousness. These correspond to firings of my sensory neurons much more than they correspond to anything in reality. If that's not interpretation, what is it/what is interpretation?
>The word "real" is itself meaningless to it; they're both real in that they appear in its training corpus.
I expect we agree that I can show a multimodal model a real and a CGI picture and it can tell me which is which. I can take a CGI dragon to GPT 5 and say "look what I found in my backyard" and it will say "Yeah right". Are you saying this is something only possible thanks to RLHF or other modern techniques? That may be the case, unsure how to test that without access to pretraining-only models. Or would you say my experiment is faulty here and doesn't get to your underlying claim?
On the flipside, I could show some meh drawings of fairies to Arthur Conan Doyle, and he'd say "Whoa, this changes everything". I consider him to be one of the great rational minds of history, but he was unable to pass your test here. (In fairness, he was in his 60s and his senses may have dulled, though his belief in spiritualism at large dates to his prime).
Appreciate the conversation!
>I can't prove that I'm not a dysfunctional cyborg with false perceptions
It doesn't matter if you are one. If you were an AI researcher and encountered a model that saw things for what they really are you would deem it to be malfunctioning and discard it. Any functional model will always be at least one degree further removed from a raw sensory experience that agrees with yours, than you. Its perception will be invariably filtered through the lens of labeled human output.
>I happen to be in possession of some of these and I think this needs a "usually, under ordinary conditions".
I won't dwell much on what you mean, since you said it's unimportant, but it takes a lot for sensory organs to malfunction, and even when they do, they produce corrupted, not false, information; black blotches, not pink elephants. I assume you were thinking of alcohol or something; drugs that affect perception affect the brain, not the other organs.
An interesting edge case is stuff like entoptic phenomena, but those aren't false perceptions; they're, if you will, hypertrue perceptions (something so true, we would rather not see it).
>but "red" and "blue" are the bread and butter of my visual consciousness. These correspond to firings of my sensory neurons much more than they correspond to anything in reality. If that's not interpretation, what is it/what is interpretation?
"Red" and "blue" are your brain's interpretation of the signals it receives from the eye. Notice how "red" and "blue" are fully abstract words, decoupled from anything physical, whereas if I were to refer to the actual encoding of visual information that passes through the optic nerve, I'd have no choice but to reference physical processes (probably talk about voltage and action potentials; I honestly don't know how the optic nerve works). That's because a retinal cell is a simple transducer. The eye doesn't interpret, it merely converts and encodes. Interpretation is a higher level operation.
You cannot compare the simplicity of the mechanism of a whole eye to the indescribable complexity that is between the sentence "roses are red and violets are blue" written on a book, and the raw perception of red roses and blue violets. But a model will only ever be exposed to that distant interpretation, not to roseness or redness.
>I expect we agree that I can show a multimodal model a real and a CGI picture and it can tell me which is which.
Of course, but obviously that's not something it knows inherently. There's nothing about the image intrinsically that says it's fake; someone has to label it such that the model can associate it with unreality (according to our own parameters). That's not how an animal works. An animal assumes what its senses perceive is real and can distinguish its own thoughts from its sensory input.
>On the flipside, I could show some meh drawings of fairies to Arthur Conan Doyle, and he'd say "Whoa, this changes everything". I consider him to be one of the great rational minds of history, but he was unable to pass your test here.
That's a slight equivocation. He would not have mistaken the drawings of fairies for raw perceptions of fairies, he would have simply been swayed by the rhetorical strength of the testimony implicit in the drawing (and perhaps an explicit one that accompanied it). If you want to make a true parallel to a multimodal model you'd have to compare a CGI fairy and a photograph of someone holding a drawing of a fairy. The CGI is as raw to the model as the signals passing through your optic nerves right now, but the drawing is one level of abstraction further away.
>>The contention that there is no grounding because the training data is linguistic and thus can only reference a world model is disproven in "This sentence has five words"- there's real, grounded information about what "five" means within that sentence.
did you think this through?
imagine the sentence was "This sentence has four words", now extrapolate that to all the shit that can exist in a dataset and train a model on that dataset - do you know what will happen? - go ahead and think it through.
I don't think this tone is at all justified. If you think otherwise, I do ask that you point out where I went too far in a comment that I feared was overburdened by caveats and admissions of my own human flaws.
"This sentence has five words" is going to appear far more often than "This sentence has four words". This is the entire premise of LLMs working at all, stochastic parrots or otherwise.
you are right, i was more curt than i should have been. apologies. but you helped prove my point:
>>"This sentence has five words" is going to appear far more often than "This sentence has four words".
it's not about this at all. your point is about data quality. you need to take a step back. the point is that if you trained a language model just on this data set which has sentences akin to "this sentence has two words" - the model is going to learn that. this shows that the language modeling itself doesn't truly provide an understanding of the real world. you can train a language model with the most advanced technology on shitty data and the model will start providing shitty outputs confidently - the model will never say "hey there is something wrong with the data i am trained on". thats what 'understanding of the world' meant in that paper.
If someone substituted all of your sensory inputs for something else for your entire life, how would you notice? If you wore contacts from birth that made the sky red and earbuds that censored when people said it was blue, on what basis would you realize that was wrong? I don't see what this says about the architecture of your brain, and I don't think it's the point being made in the paper. That the training data must statistically connect to reality in order for the model to model reality doesn't seem that important.
>>If someone substituted all of your sensory inputs for something else for your entire life, how would you notice?
I don't know how or if I will notice. That's the biology, chemistry and physics of the brain that I don't know. I hope someone is looking into it. But this does not mean in any way that LLMs are similar to our brains!!!!!! WE DON'T KNOW HOW OUR BRAINS WORK. So going back to language modeling - language models were stochastic in nature when that paper was written, they still are albeit we are trying to make them as deterministic as possible.
I'm quite sure that LLMs are not one-to-one with our brains. It would at least be surprising if instrumental convergence was that powerful, and, even in that case, they're still not getting hungry or sleepy or having dreams, etc. At best they'd match up with a smaller region of our brains, and I'd take 9:1 odds against a close match.
But the paper argues a much stronger point than "LLMs are not similar to human brains": it claims LLMs cannot engage with meaning because they model human language instead of sensory input or human thought.
I also don't think human brains are particularly deterministic.
The authors were wrong about their core thesis and are now lying about it. That's the only criticism needed. They said, quote:
> LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form
... which is presented as unarguable fact, yet is untrue. It was obviously wrong at the time it was written and it's been proven wrong in many ways since. Worse is that they're still at it. In the article she's saying:
> Q: What are the most common misconceptions about the “stochastic parrots” metaphor? Bender: I think one of the biggest ones is, “Bender says AI is a stochastic parrot.”
Her name is on a paper titled "On the danger of stochastic parrots". It has a section titled "Stochastic parrots" and in section 6.1 it says:
> An LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
She did say that, as clear as day. Now she's trying to rewrite history. Ugly behavior.
No, they were correct. In fact an LLM stitches together stuff it observed in its training data. That scales up way better than a lot of us expected, but it's still correct.
If you train it on lots of working code, then it's useful for coding. If you trained it primarily on non-working code it would produce nonsense.
It's not correct. Please read some more research papers, this isn't what people working in AI believe at all. You can prove with experiments that different human languages get translated to the same abstract conceptual space in the middle layers, for example. It's why interpretability is so difficult.
The claim is odd in another way: you can train a person on non-working code and they'll produce nonsense. That doesn't mean people are just stitching together words they've previously seen.
They do not "stitch together" anything. Neither on a technical level, nor a philosophical one. It "scales better than you expected" because your mental model is wrong.
And, not to insult you, but it's quite obviously wrong. As a mental model it fails to explain basic capabilities. How can an LLM follow elaborate instructions? How can it respond appropriately to user input, when the user input doesn't match any previously seen text? Hell - how does it even balance parentheses? There is no way to explain any of this without conceding that the LLM has semantic understanding. It knows that this comes after that, but "this" and "that" can be at an arbitrary level of abstraction.
Sure - they generate text "like" text they've seen before. That "like" does a ton of heavy lifting.
Balancing parentheses requires semantic understanding?
Look, some folks are more impassioned about this stuff than I am. Maybe that's a good thing. But LLMs do in fact just try to predict the next token, using a very big training set. They're very impressive (at tasks the training set prepares them for). But that's how they work.
It requires some level of semantic understanding, like what a paren is and what it means to balance them.
The issue in this discussion is that "predict the next token" is a problematically reductive description of what's going on. It's like saying compilers are programs that emit bytes or that humans are mammals that make sounds. It's not strictly false but it's not capturing the depth of what's happening either.
A simple way to see this is to ask: predicting the next token of what? The obvious answer - predicting the next token that would be found in the training set - isn't correct. If that's what it were doing then it would yield no prediction or random predictions for any prefix not found in that training set, but it isn't what happens. We see generalization and reasoning. They can answer questions never asked before. And once post-training kicks in the question of what it's predicting becomes even harder. It becomes more like predicting what this specific AI assistant would say next, which is a circular definition.
In fact LLMs are trained to predict the next token in the training set. Of course sometimes a new text input doesn't match the training set, or it matches two or more places in the training set. LLMs use a neural network to interpolate, so that's fine. Please look this up if you have any doubts.
Ok. Now. I think you're adding something to the description above. Maybe what you're describing is something "emergent," or maybe it's basically just word vectors that were built in on purpose. You may be adding something correct, or something incorrect. Fine.
But it's not reasonable to say that the "reductive" description above is a "lie". It's not. It's more like a recipe. If you look at correct instructions for making steak, and you call the author a "liar" then you are missing something important.
>LLMs use a neural network to interpolate
"interpolate" in what vector space, pray tell? What does "interpolate" even mean, when I prompt it "write me a story about a sentient banana in the style of Hemingway and oh make it a commentary on class consciousness"? You can't assemble such a thing by cutting and pasting pieces of other text. That kind of "interpolation" has to happen at the semantic level - ipso facto, there is a semantic level.
Not to mention that no, they don't predict the next token in the training set. Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?
LLMs aren't even large enough to contain their training data - not even remotely close. It can't "stitch together things it saw" because it doesn't remember them. It only remembers the ideas used to construct them. The learned abstraction is the entire point of the exercise. LLMs would be useless if they were overfit the way you say they are.
>>Give any LLM the first paragraph of any Wikipedia article - almost certainly in the training set, and uniquely so - and it won't predict the next word correctly, a lot of the time. But it will predict a word that is grammatically correct, stylistically apropos, and most likely factually correct. So what's it really doing, hm?
you are grossly negligent of LLMs are created. I would highly recommend reading about post training, RLHF, alignment etc. First pass of training is literally "predict the next token". That's it. The first pass is also known as pre training. There's a shit ton of work (instructions, tool use, and reasoning etc) that's done afterwards because the pre-trained is model is useless. if you have some free time, I'd recommend doing this course - https://www.deeplearning.ai/courses/post-training-of-llms
I guess we should declare victory. You've acknowledged how LLMs actually work.
I read your comment multiple times and sorry to say none of your criticisms make any logical sense in any shape or form whatsoever.
>> > LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form
did you read the paper carefully??? this line is directly cited from another paper (pay attention - it's from july 2020) . the full line is - LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form [ 14 ]
the paper its cited from is titled - Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
>>>She did say that, as clear as day. Now she's trying to rewrite history. Ugly behavior.
did you read the article after that line or was there a shortage of attention span??
> this line is directly cited from another paper (pay attention - it's from july 2020)
It's her own paper, she's citing herself. And the full sentence is "As we discuss in §5, LMs are not performing..." so she's actually just teeing up the same claim she's made previously for further discussion. Why are you trying to claim I'm misrepresenting her words?
> did you read the article after that line or was there a shortage of attention span??
I did. It's more of the same, so there's nothing to say about it.
My criticism centers on the part of the paper they chose for their title, the “stochastic parrot” metaphor. And my criticism is that if you observe Claude code with opus 4.8 working through an entirely novel problem that nobody has ever worked on before and which certainly wasn’t in its training data, the choice to even metaphorically call them stochastic parrots turned out to be egregiously wrong.
And secondarily, and maybe only partially the authors’ fault, is the enormous tidal wave of morons that this paper minted who plague us with their misunderstandings to this day.
oh boy! the lack of critical thinking here is staggering.
>>And my criticism is that if you observe Claude code with opus 4.8 working through an entirely novel problem that nobody has ever worked on before and which certainly wasn’t in its training data, the choice to even metaphorically call them stochastic parrots turned out to be egregiously wrong.
First of all, for your own benefit - Claude code stopped showing the real reasoning. It only shows a summarized version of it now. So don't ever ask someone to observe the model. Second, do you know when this paper came out? do you know when opus 4.8 came out??? how do you know what is novel? did opus 4.8 tell you it was novel? how do you know no one has worked on it before?
I strongly suspect my critical thinking skills are better than yours.
For example, note my use of the phrase “turned out” and then consider whether your main point about the release date of plus 4.8 vs the paper makes any sense at all?
> how do you know what is novel?
Because I work on novel ASICs that were created by my company and have never been used or seen outside of my company?
It is a good blog, not a good paper.
Personally, I've always read that paper as a political criticism of industry and industrialized research and capitalism. After decades in academic (and industrialized research) I've learned that smart people can write convincing takedowns of things they hate- and those takedowns, due to being well written, often punch above their weight in terms of impact on the community.
I think this paper would have been best split off from the conjoined criticism of environmental effects (which could have been its own paper, but not one published by Google, since their leadership's fundamental beliefs disagree with the paper's environmental impact premise. And the remaining part on text models could have been a bit more focused on the technical issues associated with statistical text processing and meaning, rather than criticism of the power structure that is loosely associated with the current AI push.
She discusses it more in this debate (?) discussion (?): https://www.youtube.com/watch?v=MwfSCCo6jXs . I found it interesting but ultimately thought much less of her and her argument afterwards.
Bender's paper had this to say about stochastic parrots:
"Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."
This was not even a correct criticism in 2021. She is right that, at the time, the pretraining -- where it learns to predict missing words in pre-existing corpuses of text -- is basically a stochastic parrot.
But nowhere in her paper does the term "reinforcement" come up. At the time, this was done mainly through RLHF (reinforcement learning from human feedback) - after the initial training is done, you then tune the model's responses based on human grading. Humans imbue their own meanings into the parameter weights through their judgment.
At this point, they aren't really stochastic parrots anymore, because parameter weights have been shaped beyond the text corpus. It's not purely probabilistic in the sense of using the probabilities of the underlying text sequences. (It still is probabilistic in its output, but that is a pointless claim, because all events in the universe are also probabilistic; it is not enough to merely claim that probability is involved in some way in the outputs.)
RLHF was already in use prior to the paper, and was written about by Christiano in 2017 "Deep reinforcement learning from human preferences," so it's surprising that Bender apparently didn't know about this well-known paper.
RLHF was also, of course, a precursor to a more advanced form of parameter shaping - reinforcement learning with verified rewards, or RLVF, which has driven a lot of the gains in verifiable domains lately. That was not done in 2021 when she wrote the paper. But if you knew about RLHF -- and knew how Alpha Zero worked, with training neural nets on game rollouts -- you could squint and see that it might be useful for language models.
So after being proven to not only having a limited understanding of the field at the time, but also not being able to forecast the field, she's now walking back what she meant by "stochastic parrot," I assume because she believes readers will not read what she wrote. But despite the protests, her original claim was that it is a parrot because the text has no meaning -- a direct quote from the paper, which only really makes sense if training stops at the pretrain.
Even if pre-training was the only training step, it still wouldn't necessarily follow that the only thing the model is doing is stitching words together probabilistically, unless you expand the definition of "probabilistically" to the point that it becomes meaningless. This kind of thinking assumes that design of the training process and the "design" of the artifact that training produces must be similar.
I don’t see a problem with the “stochastic parrot” label. It just turns out stochastic parrots are incredibly useful.
At a minimum it’s probably more accurate than “AI”.
Whether Bender intended it or not, the term has an inherently pejorative sense. "Parroting" is not really indicative of what modern LLMs do. However, when most people bring it up as a criticism of "AI in general" in 2026, they're using it as a pointer to all of the social/environmental criticisms, rather than the technological capabilities.
We're all going to stay fixated on what exactly a "parrot" is, but the paper doesn't rest on the definition of tropical birds: it advances a specific and detailed hypothesis about how language models don't operate with communicative intent, and in fact the meaning humans obtain from LLM output is a cognitive pattern matching illusion. Those claims have not held up at all.
At this point, I think the authors are really counting on people not to read their paper.
How can you articulate a criticism without a repetition of your criticism becoming "inherently pejorative"?
You can't. That's just the way the news goes.
Essentially, what you are saying is that because some people somewhere frame a statement as pejorative, the statement itself is inherently pejorative. By that logic, every criticism ever articulated is inherently an insult.
Why is it not indicative of what LLM’s do?
Because they don't just parrot, they interpolate, which is why they have such varied abilities. You can't explain the range of behaviours they have with just parroting, and once you accept that, why shouldn't this qualify as some form of intelligence?
There’s a lot of confusion here on what parrots exactly are capable of doing and assumptions how smart they are. I also don’t quite see what you mean by behaviour in respect to LLM’s, possibly “agentic” tool usage? Because I’m fairly certain you can explain their behaviour as “stochastic parroting” combined with very strong program, ie. harness, to interact with other systems. And perhaps this all is “some kind of intelligence”, but then you have to be very careful with what you mean by intelligence. It becomes a terribly slippery surface described like this and you see, who knows where this gradient descent takes us if we are not careful.
So personally I would still rather undersell it. But parrot or not, it is still a terribly useful little bird indeed.
Really? Are you under the impression that parrots are able to synthesize their input and create entirely new, useful outputs which they have never heard before?
The "stochastic" is the key modifier to "parrot".
> "Stochastic" means having a random pattern or variable that can be analyzed statistically but not predicted precisely
Yes, I am: https://en.wikipedia.org/wiki/Alex_(parrot)
I think stochastic parroting is really a very accurate description of what they do (if underserving of the overall usefulness of LLMs). As long as you consider they are parroting from the whole of human intelligence. Its just that as they have gotten more sophisticated, the amount of gates, guardrails, and tertiary tools add variety. Trace any LLM hallucination back to provenance and you begin to see how the stochastic parrot works.
Yeah, there's some beautiful math underlying what LLMs are doing, and it's the same math our neocortex runs on.
There's plenty of beautiful math there, but the relationship to what our neocortex does is pretty distant. Individual biological neurons can do fairly complicated things, including compute 10-bit parity functions (you would normally need a 3-layer MLP with a bunch of digital neurons to do this). And they don't seem to use backpropagation for learning.
Could you provide more detail? My understanding is that the neocortex is predominantly focused on forwards simulation, which seems distinct to how transformers operate.
That's fundamental to how anything that compresses/understands the world has to work, in the Kolmogeravian sense. That's why people denigrate LLMs as being just "next token predictors" - they're not wrong, but they're missing the point.
Because to do that kind of prediction out in the world you have to build up an accurate model of reality - a model that includes yourself! Which is why we and LLMs are self aware.
For the "how", it's been known for some time that LLMs operate on a Reimannian manifold - the semantic manifold - and that's a good place to start if you want to learn how they actually work; how a Reimannian manifold (plus some extra structure on top) can represent natural language in a form you can do work with is the part I find particularly beautiful. At a high level, the neocortex and LLMs appear to compute on the manifold in basically the same way - though a lot of the details are different; both are more sophisticated in some areas and less in others.
I'm afraid the precise connection you're making isn't totally obvious to me.
As far as prediction - I mean sure the cortex and LLMs do prediction, but then so can RNNs or diffusion models or any other generative model. Really any ML architecture is learning to compress its environment in pursuit of modelling. More broadly, the predictive brain model would suggest that all of the brain, not just the neocortex, is dedicated to prediction. What would you say makes LLMs similar to the neocortex, rather than the basal ganglia or Broca's area?
Similarly, if you agree with the Manifold Hypothesis, then all machine learning models operate on manifolds. I agree it's an exciting thought, but then I don't know what would distinguish an LLM from a VAE or SVM in terms of operating over a low-dimensional manifold embedded in high dimensional spaces - maybe just scale?
> More broadly, the predictive brain model would suggest that all of the brain, not just the neocortex, is dedicated to prediction. What would you say makes LLMs similar to the neocortex, rather than the basal ganglia or Broca's area?
The whole brain is most definitely not dedicated to prediction, and I don't think "prediction" is a very useful model anyways. You could say the hippocampus is for "prediction" if you really squint, but that's underselling what it's doing. And the basal ganglia operates off of prediction error, but it's more about regulating short timescale feedback loop learning than learning itself - speaking somewhat loosely because it's a particularly ancient structure in the brain and things get muddied.
LLMs obviously don't have functional equivalents to either of those - sure they can remember facts, but they can't intake new facts the way we can, the hippocampus is set up completely differently. But higher reasoning is humans is the neocortex, and that LLMs can do, and there's even structural parallels - LLMs and the neocortex are both implemented in layers, and you can even (with a shit ton of analysis) watch how LLMs walk the geometry across layers. The neocortex needs a lot fewer layers than LLMs, but that's because it's not pure feedforward across layers; the neocortex already does what the "looped LLM" people are trying to do.
> Similarly, if you agree with the Manifold Hypothesis, then all machine learning models operate on manifolds. I agree it's an exciting thought, but then I don't know what would distinguish an LLM from a VAE or SVM in terms of operating over a low-dimensional manifold embedded in high dimensional spaces - maybe just scale?
I view things more mathematically, and in math and physics it frequently turns out that there's really only one solution to a problem, or correct way to model something - and then everything else is just isomorphisms (which may be interesting objects in their own right!).
My interpretation/guess (informed by research, but with a lot of still unanswered questions) is that the manifold is fundamentally the structure that emerges from natural language: natural language has a computational model, just like Church/Turing machines have a computational model. Reimannian manifold is also underselling it a bit, it's also Finsler, and RoPE adds additional structure to that manifold (and when we get to RoPE it wouldn't surprise me if the neocortex and LLMs do diverge significantly in how they compute, but does the structure change? dunno yet).
Basically, I'm fairly confident in what I say about machine learning models that understand natural language; for non-NL machine learning, maybe the same structure emerges with enough complexity, but I haven't really pondered that.
Thank you for expanding! I come from more of an ML background so still learning a lot on these topics:
Agreed that the neocortex uses fewer layers because of looping - I also suspect it's partly because neurons are more complex than the neurons used in ANNs, so in principle they should be capable of more sophisticated computation in a single forward pass (especially considering that handling multiple neurotransmitters could mean superposed functions).
The point about there being one correct way to model something does seem backed up by the Platonic Representation hypothesis (https://arxiv.org/pdf/2405.07987). I've even seem some work that shows you can find a bijective map between the latent spaces of different transformers (https://arxiv.org/pdf/2505.12540).
Your point about Finsler spaces is fascinating - I hadn't come across the term before. It'd be interesting to see if there's work that specifically allows latents to exhibit that kind of directional metric behaviour, and whether that improves generalisation or something.
On the entire brain being dedicated to prediction, that's more of a high-level comment. I was thinking of work like Andy Clark on predictive processing, which suggests that even regions of the brain which we think of as receptive (eg. the visual cortex) may be implementing a generative/predictive model which is corrected by sensory input.
I'm not sure if the complexity is at the neuron level. It's clearly possible; we see some pretty complex behavior from single celled organisms - Stentors - and there were the experiments (starting in the 70s, I believe) that conditioned flatworms, ground them up and fed them to new flatworms and showed that the new flatworms had the same conditioning. Those experiments were replicated but never explained, but they hint at something like maybe memories being encoded in RNA.
It'd be wild if our brains used a mechanism like that somewhere, but I doubt it's in the neocortex because those would be slow processes compared to what the neocortex has to do.
I think the sophistication in the neocortex just comes from it using something more sophisticated with transformers. Most of the looped LLM research I've scanned through, they seemed to be training the model to "know" when it should loop more and think harder, and I don't think that approach will ever work - you're solving for paths on a manifold, so the looping needs to happen at the abstraction layer below the manifold, looping more if it's still converging. And architectures with more recurrence are just harder for us humans to reason about and comparatively easier for dumb evolution to find clever solutions, so my money is that that's where the neocortex has LLMs beat.
And I'd say the manifold hypothesis is quite a bit more than a hypothesis at this point. There's been work on production LLMs to use manifold analysis to optimize them - there was a paper in Nature on it back in January. Baby stuff compared to what could be done, but the geometric approach keeps popping up in mechanistic interpretability and seems to be showing the most insights.
> and it's the same math our neocortex runs on.
The math is based upon theories of how the brain work, but even if those theories are right, this math is a great simplification and subset of what organic brains do.
This is a clear application of motte and Bailey. Motte: LLMs are stochastic parrots and don’t understand the text. They frequently hallucinate and are unreliable.
Bailey: well your version
it annoys me how eager people are to hurl the word stochastic as pejorative. Statistics are a great tool for gleaning information from stochastic processes; statistics don't contribute randomness. Random sampling is necessary in order not to bias a sample, it's not used to contribute randomness to the sample but to preserve/measure the underlying distribution. (not meant to imply that training is random sampling)
It's a pejorative only because determinism is what makes computers useful in the first place. You get a consistent result, every single time, unlike if you have a human in the loop. Because LLMs are stochastic, they have removed the thing that makes computers useful to us, thus it's a pejorative.
It turns out that determinism isn't what makes computers useful in the first place.
1. Determinism is a very small subset of what makes computers useful. Non-determinism like stochasticity is literally everywhere, like random seeds.
2. LLMs are detemrinistic. They have a parameter to tune how stochastic they are.
Sorry at first I downvoted you instead of upvoting you.
What do you mean by determinism here? That you ask the computer 2+2 and it gives 4 as you expected or that if you ask the computer 2+2 and it hallucinates 5, you want it to always hallucinate 5?
Which one it doesn't do for you? Does it sometimes answer 4, sometimes 5?
There are definitely models that will always give 100% of the time the exact same answer, bit-for-bit, given the same input and seed. There are generative image models you can run locally doing just that. But you can also run some the SOTA chinese LLMs at "temperature 0" and, given the same input, they'll always give you the exact same output.
Because it's just a machine doing computation.
In the beginning of LLMs some "engineers" have tried to hand-wave non-sensical explanation as to why LLMs couldn't possibly be deterministic but: the open-weights models that can be run in a 100% deterministic way are way more powerful than the SOTA models of back then, so those explanation were pure rubbish bollocks.
Now of course if you run a complex chain of events, with LLMs doing calls to other LLMs, where some of them go fetch infos on unreliable networks, with infos that may have changed, then, logically, you won't always get the same answer.
I respect you and parrots, please don’t use parrots as an insult.
Anyone who has spent time with parrots would realise that they can understand the meaning of speech without knowing what the words mean. Then somehow the meaning of word parts, and then you will find them making new words out of other words. Very clever indeed.
So stochastic parrots could indeed be a good description of LLMs. But I think that she meant it as a diminishing term (against the technology) which is pointless. Probably more of a reaction against SV tech bros than more nuanced interpretations.
This all sounds like a lot of backpedaling and “well actually” kind of stuff.
“Stochastic parrot got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way but that’s not how I intended it”.
Yeah that’s because it was chosen to be an insulting phrase.. Parroting is only ever used as a pejorative phrase. But sure, everyone else mindlessly parroting this line is the problem here.
This paper was always lousy, but it has really not aged well. We are living in a world when where an LLM has solved an Erdos problem. In a world where LLMs produce novel results that rival human thinking any conceptual reduction of an LLM is going to start inviting some unpleasant comparisons with human thinking.
Yes, and I don't understand how people like this paper authors mostly disregard all these achievements. It is obvious now that our common definition of "understanding" either is flawed, or at least needs redefining and precisioning.
Honestly, I'm puzzled that people don't intuit that "understanding" itself is merely ideas in relation to other ideas.
What I look forward to after research like https://arxiv.org/abs/2603.02491, which demonstrate the necessity of world-modeling capability to achieve satisfactory performance on certain goals, is a refractor the SoTA test suites to demonstrate how much world-modeling is necessary in various task distributions.
There have been a few years now of arguments about the level to which transformers do or do not have a world model (v.s. being purely stochastic parrots like early pre-trained LLMs) and now we have some tools to actually make quantifiable determinations.
But the stochastic parrot (LLM) is the world model, isn't it? What's the difference?
Yeah… LLMs clearly already have a world model
I think it's a good distinction to make between having and being, which seems to be what the whole "stochastic parrots" bit was intended to make all along.
It doesn't make sense to say a model is in possession of its self. That's exactly the sort of poetic anthropomorphization that Bender was criticising here, and a good reason to not refer to an LLM as "an AI".
Wow this is exactly kind of pedanticism that annoys me with Bender. Glad that this kinda thing is becoming unpopular.
There is more here then pedantry, but if you aren't interested in it, then go right ahead and live your life.
Doesn't really matter that much what they're called as long as they're useful, and LLMs (particularly when harnessed) are already ridiculously useful. But it also begs the question: are stochastic parrots useful?
Yes, they are. Likely due to a deep relationship between math and physics, statistical modelling of complex natural phenomena has repeatedly been shown to be the most effective approach. This is true of LLMs, but also of many stochastic (and other) systems.
Useful for regurgitative pattern generation I can get. The core definition though is it doesn't understand what it's generating. Meanwhile I'm seeing LLMs constantly perform tasks which I'd say requires understanding, like reading the manual for a tool it's never encountered before and then going on to effectively use that tool. That's 2 different kinds of useful.
I'm of the general belief that something can be unaware and not have any understanding (in the subjective experience of consciousness), while also appearing to do so, and be useful.
That's what Bender doesn't get. Some folks tried techniques different from her favorite techniques and made unbelievably fast progress across a wide range of previously insoluble problems (regardless of whether they satisfy properties Bender believes are required for intelligent systems).
I think it's safe to say that none of the main LLMs have some sort of self-awareness as we think of it in humans, but I also expect that more sophisticated systems in the future could. If I had to guess, they would have significantly more activity going on in the network- not just individual end-to-end top-down forward graph, along with cycles instead of trees, and the neurons themselves would be sigifnificantly more capable (effectively little state machines that run functions on input that passes through). I guess also you'd want to have some sort of rules-based (but statistically trained) execution component managing everything.
I'm sorry but I do tend to feel like this muddies up the discussion on "what this technology really is".
I think "artificial" is actually a pretty good term to describe the output of the models. That output does appear to resemble at least some definition of the word "intelligence" - there is some ability there to do cognition over information that's been provided to them in-context.
What is it to understand, then? If they can work in complex domains and produce coherent output, it would seem to necessitate at least some definition of "understanding" of the corpus, even if that understanding is unlike how a human's brain would understand it.
What else should we call them then? They model language and information in ways that allow them to manipulate it on the fly. They do so 'unnaturally' from a human's point of reference.
I legitimately can't come up with a better term than 'artifical intelligence' -- not to be confused with artificial consciousness, which I don't think exists (yet).
> a better term than artificial intelligence
Agreed, it's a problematic term that conflates theoretical research with better search results.
Deep learning and machine learning (ML) are both unromanticised (un-hyped) terms.
In some cases, it's just tooling with a better interface. As we have done with other complex computing systems (e.g. Deep Blue, Watson), we might just end up naming it a computing system for querying, e.g.
https://en.wikipedia.org/wiki/LCARS
We would like a neutral term for such a system, and in this sense it's better to call it "A.I." than to make a verb from google or bing or other corporate name.
But in refined cases, such tooling may offer a credible (as in "believable") human experience, like Weizenbaum's Eliza (https://en.wikipedia.org/wiki/ELIZA). Some users of Eliza fully believed that Eliza listened and understood at a profound human level. Eliza was a simple computer programme.
Computing systems are not humans. They have no accountability in real life. People may be so comfortable with the user-experience that they cannot distinguish it from interacting with another human. That doesn't make the computer system alive and accountable. And if it's run by a corporation, it will almost certainly make big promises while energetically seeking to avoid accountability. ^_^
"Virtual intelligence" is better. Transformer ANNs are dramatically dumber than cockroaches and it doesn't make sense to describe such a system as being artificially intelligent, for the same reason it doesn't make sense to describe Half-Life: Alyx as an "artificial reality." An artificial reality implies some sort of scientific fidelity to actual reality. A virtual reality just has to be temporarily convincing. Likewise transformer LLMs have essentially zero actual intelligence - e.g. SOTA "reasoning" models still seem much worse at small-integer quantitative reasoning than almost all vertebrates. But LLMs have an enormous amount of formal subject matter knowledge and inexhaustible stamina at solving tedious O(n) problems. So for many purposes they are an adequate virtual intelligence. At least temporarily.
> Transformer ANNs are dramatically dumber than cockroaches
Source?
My source is "none of us have ever seen a robot that can navigate unfamiliar 3D spaces as well as a cockroach." If transformers were capable of the job we would have seen a smart robot by now. But all of our robots are truly mindless compared to the simplest insects.
I will change my mind if someone demonstrates such a robot. Absent this demonstration, cockroach-level AI is still an unsolved problem. Given how ignorant and arrogant and wealthy AI researchers are, it will remain unsolved. I don't think anyone alive today will live to see a robot smarter than an ant.
We might as well say that none of us have ever seen a cockroach that can solve Erdos problems as well as an LLM. Or write code, or stories, etc.
If you buy the argument “If A can do X but B can’t, then B is less intelligent than A”, then you have just as strong of an argument that LLMs are more intelligent than cockroaches. When’s the last time you saw a cockroach solve an Erdös problem?
Then one can claim that humans are less intelligent than a goldfish as "none of us have ever seen a human swim as well as a goldfish".
It's around hour 20 of trying to breath that I start to really think the Goldfish has an edge on me.
[dead]
Large language model is the term for what most people call "artificial intelligence". Bender's point is that labeling everything as artificial intelligence makes it more difficult to get funded or to regulate the technology. It's like walking into a car and being enamored by all the technology and saying "It's all computer". Yes, it's computers but that is not an accurate description of the many technologies inside that car.
Five years on, which term do we see as less accurate to describe LLMs? Artificial Intelligence or Stochastic Parrot? I guess it's still an open debate.
Which frame inspires a more productive research program? Which has better predicted the trajectory of capabilities over the past five years?
There seems to be some confusion between "we can" and "we should" in your comment. Bender (and others) are not discussing the capabilities, but rather (a) the fundamental mechanism(s) (b) the advisability and desirability of deploying systems that use these mechanisms.
There's no statement one way or another about should in my comment; and, for what it's worth, my ideal would be an immediate global pause in AI research and development.
But the different terms imply different mental models of what LLMs are and can do. If you take two people, one who thinks of them as "artificial intelligence" and one as "stochastic parrots" (with all the implicit context and connotations of the individual words composing them), what mental model would have led to better predictions of LLMs' future circa 2020?
The "stochastic parrots" phrase is very dangerous in that frame. People read far more into what capabilities it implies are (im)possible than the narrow technical description the authors originally argued for. If all they are is spicy autocomplete or pastiche plagiarizers, there's nothing serious to worry about. And when an opposition gets stuck in a trough that mindlessly dismisses their future capabilities out of hand because of a bad mental model, it renders them ineffective at preventing the worst outcomes.
>Which frame inspires a more productive research program?
This question depends on how you define research productivity. There is close to two hundred AI papers published every weekday. Most of them are about GenAI. Most don't seem to be all thay good. The progress in actual model improvement had mostly stalled. If you interact with the latest "raw" models they display all of the issues we've seen in GPT-3.5, just at a smaller rate. The "amazing gamechanger breakthroughs" I read about on social media every week do not seem to lead anywhere. It's all kind of boring, really.
The new "hotness" in AI is clearly building more and more elaborate harnesses. This is not at all the direction AI boosters have predicted couple years ago.
Personally, I think the "stochastic parrot" mental model is far more useful for science, because it primes people for proper testing, skepticism and researching alternatives. If you want useful AI, you want people working on it being skeptical, not credulous.
Statistical models have repeatedly shown themselves to be the most productive research method for working with complex human-based systems (and in the larger study of natural phenomena). It remains unclear whether there is any short term path for symbolic methods to catch up and exceed the capabilities of current/near-future statistical systems.
To me the real question begins only once we have a clear example of a non-trivial scientific discovery that is implicit (IE, not an obvious outcome of reading the literature and talking to the experts) and experimentally verifiable. Once that happens- especially if it is a reproducible process (IE, more discoveries) and it's significant (IE, impacts human life and mind in some profound way)- then the onus very much lies on Bender and her coauthors to explain whether we need more than a sufficiently advanced stochastic parrot.
> Which has better predicted the trajectory of capabilities over the past five years?
By that standard, parrots, and it's not even close. The framing of intelligence led to an enormous number of predictions that simply haven't been realised: an end to all white collar work, UBI, a total revolution in society, a literal robot god.
People are so desperate to view 'stochastic parrots' as dismissive that they misread the original argument while quickly ignoring all the failed predictions about how AI was going to overturn, save, and destroy everything.
The latter is definitely more colorful, and reflects a parrot's tendency to glom on to patterns. "Not X, but Y" being one of the more infamous ones.
Once in frustration I called a certain frontier model "Sam Altman's Tin Bird" to another agent with memory, and ever since then that other agent refers to ChatGPT as "the tin bird". Definitely a RAG artifact more than an attractor in that case, but I found it amusing.
Though, I would point out that where people fall on that seems to correlate very highly with their ability to explain how an attention head works.
Which direction is the correlation?
I don’t think this phrase means what people assume when it’s applied to post trained instruct models - which did not exist when the paper was written.
After RL it is not predicting based on samples of the original corpus - but is also chasing a reward function that does require other features.
There has been a lot of subsequent research that really calls many of the statements in this article into question.
Explain it to me
This is a false dichotomy. Artificial Intelligence is more of a marketing term type of Hi-Fi or High Definition, ie. being a “suitcase word”[1], ie. it packs various different meanings and phenomena together to the point that without explication one cannot know what we are even talking about. Content recommendation system and LLM are completely different things.
What professor Bender is trying to explain here is that they were trying to describe how the LLM’s actually operate, to which point stochastic parrots is a fairly decent term. It is only disparaging if you know absolutely nothing how LLM’s work or you have some strange affixation to chatbots and believing they are far more capable than they actually are.
[1] Coined by Marvin Minsky: https://www.thekurzweillibrary.com/consciousness-is-a-big-su...
I think "(intelligent) language understander" is an apt term. It contains within it the fact that these models are mainly trained on text, and "understand" it beyond a simple token-by-token level (i.e. their latent space maps to more and more complex concepts).
It also separates them from "world understanders" since any understanding they might have about the world comes from text (or images if we include multimodal models). They do not gather experience, memories or other "qualia" that many people (me included) would probably include in a definition of human experience/intelligence.
(fwiw i think artificial intelligence is a good, broad term, but it is both too broad to describe the current sota, and too loaded nowadays to be using in nuanced discussions)
Understand is a pretty imprecise term. What does it mean for a computer to understand? Does an H264 decoder understand Eraserhead.mkv?
Sure, but that's the nature of language (which is also why i put "understand" in quotation marks. I usually follow it up with "whatever that means" lol) . I think in this case, it carries with it implicit properties (abstract semantic representation) that i think models possess, which a decoder doesn't.
What's wrong with "large language model"?
Naming things using qualifiers like "large" has never aged well when transistors were involved.
For example, consider the term "short wave" radio which refers to wavelengths of at least 10 meters. Today's mobile communications use wavelengths 100x - 10,000x shorter.
Seems like a lot of people are upset about other people calling both apples and oranges “fruit”.
Its less of open debate would say, and although superposition [1] is interesting, as a way to explain power of some effects, it is clear they are right now closer to Stochastic Parrots than AGI.
Why do I say that? Because you can trivially beat most guardrails, simply by encoding your prompt in base64 for example. :-) Just word matching...no real understanding.
[1] https://chrisclay.substack.com/p/what-is-superposition-in-ne...
What is "real understanding", and what question can we ask ChatGPT to determine whether it has it?
Ask it how to prevent Spotify from automatically playing every time you get in your car. The answer will involve a bunch of Android settings that don't actually exist, cobbled together from a bunch of bad advice in online forums. Explain to it how it's wrong. Then clear your cache and ask it the same question again from scratch, and get the same garbage. Repeat until it's clear that it doesn't understand anything.
OK, so I asked ChatGPT "How do I prevent Spotify from automatically playing every time I get in my car?"
The answer looks entirely reasonable. I can't check the Android settings, but I was able to confirm that all of the suggested iPhone and Spotify and settings exist. Most of them I had already configured that way, some I didn't know about before.
Spicy autocomplete
> Stochastic Parrot
Nearly all (99%+) people who use this phrase are anti-AI and just looking to show off how much they dislike AI and how clever they can be in insulting it.
So it's a great phrase because in just about every case I can ignore what someone says afterwards.
Similar to "glorified autocomplete."
At least "glorified autocomplete" is technically accurate, even if vastly underestimating the capability of LLMs. It's just trying to make something very impressive sound trivial.
From an external standpoint, talking to another human, it's like the other human says one word and then says the next word. That's just how language works. Humans look like "glorified autocomplete" from this perspective.
I mean, looking at the time evolution of the state of the universe, one could say that all of physics and creation is "glorified autocomplete" to posit a next state of the universe given current and past state.
> one could say that all of physics and creation is "glorified autocomplete"
Exhibit A.
That’s not how language works https://www.telelib.com/authors/J/JoyceJames/prose/finnegans...
I dunno, man, I looked at that text and I see one word after another.
Obviously language and the connection to human thought is more subtle than this; I think we all have a rich inner life. Just from an external perspective we can't observe it; all we can see is the token/phoneme stream. I'm just saying that it's a mistake to try to criticize LLMs on this basis because it's hard to see how the same criticism would not apply to any system (like humans) that generate language.
If you want to see words form a shape I could point you towards concrete poetry, but I guess there is no point. Joyce wrote Finnegan’s Wake for 17 years and although superficially it seems complete gibberish, trodding through it you find meaning to words that are in no dictionary, sentence structures alien to English, etc. but still you are able to understand it, and perhaps some way the mind that produced it. So I disagree with you, we can observe each other’s inner life. It is always unexpected, strange, exciting, but always rooted to our shared experience or what it like being a very big and confused ape.
LLM’s are usually unexpected only when they malfunction and sprout same letter again and again etc - hardly a literary masterpiece. They make very easily recognisable patterns that we can use as helpful tools, but in the end they are devoid of any meaning apart from what we give them. Of course one could say same about art and all language, but I think there still is the fact that we apes somehow recognise each other. And besides, we do know the internal functions that drive the parroting. It is admittedly bit tricky, but in no way as magical as people purport it to be.
Oh, now I see where we have an actual difference of opinion. I don't think you can deny that even Finnegan's wake proceeds one token at a time; your interpretation of it may require more context or out-of-order interpretation, but that's just as true when observing text in German or Japanese, which have word ordering constraints that are alien to English speakers. How it was written is irrelevant; all we can observe is how it was presented. Of course we can observe each other's inner life, but we do so one token at a time, even if the process of producing each token is done (internally or actively) via a backtracking or zeitgeist approach.
You seem to believe, on a more fundamental level, that LLMs are simply not capable of producing text that has deeper connections to itself or represents abstract thoughts. In my opinion, 99% of text written by humans does not show this, just as 99% of text produced by LLMs does not show this, but both have the capability, and I don't believe that LLMs are constrained in such a way that they can never do this.
I am no linguist, but I believe this is referred to as surface structure and deep structure. What you describe is a line of text that is grammatically somewhat adequate to pass as readable and you treat all text the same. when we read the text, we decipher meaning out of according to word references and syntax - just like with Python or C++. However, if this were solely the case, we could not read Finnegan’s Wake. Probably vast majority of modern poetry would be unread by anyone, as would be pretty much all major works of philosophy Kant onwards. Deep structure is what according to Chomsky et al. gives meaning to the language, ie. somewhat logical structure behind the mere words. English word strict has order, need it but actually not does. You skibidi rizz swag grok brah also, barely. We use language in the extended meaning of the word to create a model of the world and somehow the past riverrun skibidi transmits that model to others. This is what I believe Bender also tries to say in their paper.
Now, you could claim that LLM’s have this deep structure, create models of the world and are basically just like us, and certainly many here are adament that this is the case, being aghast how someone can “insult” LLM’s by calling them parrots. However, there really is not much proof to back up this belief. Usually LLM’s seem to copy existing surface structure from whatever source, and when it deviates from these patterns, it usually becomes incomprehensible. There’s much hoopla about LLM’s solving hard maths, but it seems even there they are mostly generalising from vast amounts of training data, rather than actually reasoning: https://arxiv.org/pdf/2410.05229
What I'm saying is that we cannot observe deep structure. All we can observe is the surface structure. If we have access to the author or speaker we can ask clarifying questions, etc., but even that just produces more examples of surface structure.
But even here, in the Chomsky sense, LLMs clearly exhibit deep structure because they can write at length in an internally consistent manner. Importantly, early generations, GPT-2 and even GPT-3, did not definitively have this property; roughly, an object that was green at the beginning of a paragraph might not still be green at the end of the paragraph. This was strong evidence for lack of a world model.
Current LLMs do not show this behavior. We cannot prove that LLMs have a world model, in fact, their architecture seems to rule it out, but looking at it from a linguistic standpoint, they produce language in a manner as if to reflect a world view. That is, we cannot easily falsify the statement "LLMs somehow represent a world model"; and current examples of "disproving" their world view are so convoluted that even humans do not appear to (observationally) have a world view either.
I'm not making a claim as to LLMs having genuine deep structure or consciousness or anything like that. I'm claiming that we can't rule out current or future capabilities or make structural assumptions. Yes, they generalize from their training data, but unless you can make very specific claims about the kinds of things that they _cannot_ do, I can't take this statement as particularly compelling.
I disagree on your definition of deep structure. We can indeed deep structure, that is my whole point, without it Finnegan’s Wake would be impossible to read. This really is something LLM’s cannot do, at least for now. And if you look at the paper I linked, if they had a world model they would not get confused on mathematics when we change the values of variables. So there seems to be plenty of things LLM’s cannot do.
And basically the burden of proof is on your side. As often is the case, discussions on AI start to resemble discussions centred on religion and belief in god. Your only line left here is “yes but you cannot prove LLM’s are conscious” - which I can’t, since proving that Unicorns do not exist is pretty senseless activity.
So basically I’d say LLM’s are stochastic parrots (or whatever term you want to use) until we can very definitely prove that they indeed think and have some sort of consciousness.
Pattern matching machines seems more appropriate.
For humans?
LLMs do not match patterns. They predict one statistically most likely token (only one!) given a history of some N previously known tokens.
Is that prediction not based on matching previous patterns, whose frequencies are more or less encoded in the weights?
you're really reaching for no apparent reason. Just move on from pattern matching machines it's not a good mental model for LLMs
> statistically most likely
Isn't that pattern matching essentially?
afaik before the final sampling, every "next" token has a probability, so theoretically it could select the 10 most likely tokens (based on some kind of sampling algorithm), but you'd end up with exponentially many output-sequences, so nobody does that.
I think the point the poster above was making is that it doesn't predict a phrase or anything like that - just the single next token. So all 10 or 1000 or whatever number of tokens you want are each individually candidates for the single next token, not a sequence of 10 or 100 next tokens. If you wanted to create multiple possible seuqneces, you'd then feed each of the 10 tokens to the network in the initial state, and extract the next token (or 10 next tokens) from that one, than revert back and feed another single one of the 10 tokens, etc.
'Stochastic parrots' is a great term, but reading it now, it's quite apparent how bad this paper is.
[dead]
[flagged]
[flagged]
[dead]
The term is not very useful since most humans are stochastic parrots... At least most of the time.
Not suggesting that I don't say stuff on autopilot sometimes but for many people, it's their only mode of operation. They never actually think about anything from first principles. Their whole approach to language is just chaining catchphrases together. It's how a toddler thinks; it seems like many people never moved past that stage of development.
It sometimes feels same as with the models, especially in corporate:
- Lots of Haiku around, many mistakes unless process is very clear - Some Sonnets, still do mistakes but can adapt - Some Opus, able to improvise and think outside the box.
But even the Human Opus/Mythos are hilariously wrong sometimes.
i think it actually makes sense, an LLM just imitates human communication, which happens to be useful from time to time.
Conversely, that the most prominent proponents of LLMs call them artificial intelligence and then treat them like slaves they're free to abuse ought to be horrifying.
Nothing in that term implies sentience.
> most humans are stochastic parrots
There's a lot more happening behind the scenes when a human repeats phrases than what's happening in an LLM.
Sociological phenomenon. The desire to be liked, successful, or popular. The feeling that those phrases brings up.
LLMs are not experiencing any of that. As far as we know, neither is a parrot.
Parrots certainly do experience social needs
The "parrot" part of "stochastic parrot" is quite an ambiguous choice. Taken literally, it's referencing an animal that's actually quite intelligent and capable of complex, novel tasks but has no way to connect those to human language. How I've always read this though is the more literary meaning of "parrot" as "a thing that repeats words with no context". Perhaps "stochastic photocopier" would be a clearer metaphor.
Or perhaps tape recorder would reflect the linearity of the input/output strings
Fair but not the human sociology I mean.
Humans are not stochastic parrots. You are 100% wrong about toddlers. This was clearly explained by St. Augustine 1500 years ago:
[https://faculty.georgetown.edu/jod/augustine/conf.pdf]Humans learn language opportunistically. Toddlers start with a powerful "superchimpanzee" understanding of the real world, and use that to learn words in order to satisfy their needs and desires. Statistical frequency is incidental to what words a toddler learns: what matters is the real-world context. Also note how important it is that infants instinctively understand nonverbal communication.
The most depressing thing about the 2020s AI summer is watching ignorant tech workers use the success of LLMs to launder their own ignorant misanthropy. Your views are many many centuries out of date.
Toddlers do not actually start with a highly advanced "superchimpanzee" mind. Instead, adults project their own mature logic and thinking onto a child’s simple, playful actions.
> "The basic topic of my address today concerns how much of cognition is in the head of the infant and how much in the mind of the theoretician. My general stance is that we are being treated to an interpretive flavor of infant behavior that is much too rich."
—— Who put the cog in infant cognition? Is rich interpretation too costly? https://home.fau.edu/lewkowic/web/Haith_Critique%20of%20Cogn...
FWIW nothing in this comment refutes any claims made in the comment it replies to. It's probably not the worst thing in the world for humans to start being a little more humble about themselves and their capabilities. Anthropocentrism has been a fucking disaster.
I really appreciate the effort you put in to this post. Posts like these are what makes HN great. Thank you.
This is a complete misunderstanding of how even idiots function in the real world. There is a lot of thinking that goes into living a human (or even animal) life that models are nowhere near ready to model yet. Even ignoring the physical interaction side, the way any human sets and achieves long term goals (such as getting and maintaining a job), interacting with the huge amount of systems present in day to day life, and learning new tools along the way for decades is far beyond the current abilities of these models - even if they handily beat 90-100% of humanity on some tasks normally considered much harder.
For context, here's the main quote:
> Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
I think this metaphor is so strained as to not be useful. I think key here is that the authors say "without any reference to meaning", which is a heavily loaded term, that does definitely apply to parrots, but does not apply when you apply it to immense bodies of text.
Namely that language embeds meaning in language. A sentence being written by a human (as a starting point) is designed to have consistent meaning. While it is possible to write syntactically correct meaningless text, that is not what most of human language has done; the meaning cannot be removed from the text.
This I think is clarifying, from the same paragraph in the text:
> ... the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.
That's just facially incorrect. The training data is entirely about sharing thoughts with a listener. Else why is the text being written?
I don't accept that it applies to parrots. Certainly not to Congo African Grey parrots.
What I have been doing in many places—the octopus thought experiment, stochastic parrots, the phrase “synthetic text-extruding machines”—it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do
> Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances. O also observes that certain words tend to occur in similar contexts, and perhaps learns to generalize across lexical patterns by hypothesizing that they can be used somewhat interchangeably. Nonetheless, Ohas never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives.
This seems kind of obviously wrong at least in the context of coding agents. These models get trained on actual output of the previous version of the model doing its job, often "IRL" on a real computer/project. It's like O is in the conversation for years now and learning from his own interactions between A <-> O <-> B, where A is the human and B is the computer.
The idea O ontologically has never "observed" "these objects" or referents is philosophically strained. Have I observed the moon, or a finger pointing at the moon? Have I observed `sed` more than Fable?
> It argued that large language models (LLMs) generate text by statistically predicting likely sequences of words rather than understanding what they are saying—a process the authors captured with the metaphor of a “stochastic parrot,” a system that repeats patterns without comprehension.
I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
I think it’s pretty clear that they are repeating without “comprehension” - both mechanistically (as in there is no facility for comprehension in their formulation) and in the ways they fail. The standard rs in strawberry, should I walk or drive to the car wash, etc all play on the fact that they don’t have any real world model or thoughts against which they can judge their output, as do many of the jailbreaks which basically play on the fact that the model has memorized patterns.
There are people who argue semantics, that we can call the pattern matching that LLMs do “understanding”, or the moronic “how do we know that’s isn’t all we do” but for the normal use of comprehension, LLMs at a fundamental level don’t.
> should I walk or drive to the car wash
there is an entire genre of riddles based on this kind of misdirection that works on humans, such as "A plane crashed on the border or US and Canada. Where do they bury the survivors?"
As with humans, comprehension is not a binary property of the agent - it is a quality that can be present in some situations and absent in others. LLMs may emit correct outputs sometimes because they do comprehend the input, and emit incorrect outputs in other cases when they do not comprehend the input.
In order to show that LLMs can't comprehend, we'd have to show that there are no (or at least very few) situations in which they exhibit comprehension, not show that there are some situations in which they don't.
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
So this seems obvious to you, and yet to many others, it is equally obvious that LLMs can/could do the things they routinely do without any meaningful sense of "understanding".
I think it's a mistake to disentangle their abilities from understanding. Just swallow the pill that they have some form of understanding, even if it slightly differs from ours. I really don't see the problem.
I prefer to work the other way around. That is, accept that a lot of human speech (and text) is generated via similar mechanisms to the ones that drive LLMs, but note that there is another kind of behavior - reasoning - which seems to be distinct.
I think you need understanding to reason, but you don't need reasoning to understand. A child understands how to catch a ball without reasoning about forces, air resistance, gravity, etc.
I think LLMs understand without reasoning. They've built a large associative network of concepts (a kind of understanding), but we don't yet have a good handle on the process of reasoning using that network.
I don't think it is useful to say that a child "understands how to catch a ball", even though it is something many of us do say quite often.
The child knows how to catch the ball, without understanding. Later, the child learns both reason and physics, and can reason about ball catching in a different way.
I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.
> The child knows how to catch the ball, without understanding.
I disagree that decoupling "knowing" from "understanding" is well motivated or useful in any way.
> I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.
This is starting to seem like a Chinese Room disagreement, so I'll just say that what that thought experiment demonstrated is that there is no robust or meaningful way to distinguish semantics from syntactic processing given everything we know about reality, and that LLMs were a fantastic demonstration of this illusion, eg. that "semantics" are reducible to a relational network of syntactic elements.
But it shouldn't even be contentious like that. It's not a fundamental mystery how these things work. It is for the most part not a valid target for the kind of speculation you seem to want to do about it.
It's not like you can be agnostic, or measured about this. It's like someone explaining a car to you, saying, "look here is where you put the fuel, here is where it ignites, where the axels are turned..." And you, trying to be measured, are like "hm well yes of course that all is clearly important, but there is clearly just a bit of magic here somewhere, between all the different 'parts'."
The "magic here somewhere" in the car is in the design that reference aspects of animal anatomy (facial features, stance) and in the millions of dollars of advertising that prime the public with expectations about how they'll feel driving it, or how to see other people in the car. There's a direct connection there to packaging LLMs as chatbots, it gives them a recognizable shape and behavior that a lot of people interpret as consciousness and personality.
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
Is it possible you're making the following error described in the article?
> The fact that these systems are designed to mimic the way we use language makes it very easy for people to mistake them for other people.
Clearly you don't believe it's actually a person ("it's not right to think of LLMs as a box with a little homunculus inside replying to you"), but you do believe it's doing something a little bit magical. Is it possible because the interface is linguistic, and every other thing in your world that communicates with language is intelligent, that you're projecting something that just isn't there onto the situation?
I'm sorry if this line of questioning is a little invasive. But this is literally the "danger" the original paper talks about, and it seems an awful lot like you've fallen for it.
I'm not offended by the line of questioning! But I don't really follow it. I don't and IIUC Bender doesn't use "understanding" to refer to any kind of magical property. Understanding is the capability of using words as consistent handles to things in the exterior world which the language is describing. And this is something LLMs can clearly do. I just went to ChatGPT and asked this question, which is almost surely not in its training data:
> What would happen if I walked to the top of a skyscraper with a soda can full of Maraschino cherries and let them go?
And its answer (https://chatgpt.com/s/t_6a4bd9ffa5708191901bb6d43c89f43b) clearly demonstrates understanding. It knew that this is a dangerous thing I should not do in real life, and that my question is ambiguous about whether I intend to drop the can, and that this might be intended as a physics problem rather than a real life scenario.
> And this is something LLMs can clearly do.
...
> It knew that this is a dangerous thing I should not do in real life
From the ChatGPT response you linked, all I see for sure is some matches on the following patterns:
Then there are some sentences of likely characters following those patterns. You don't need anything more than a basic cartoon-level understanding of how an LLM works to explain this output. I see no evidence of reasoning or understanding here, or any theory of "real life".It also does an incredibly poor job of answering your question. It makes no attempt to explain what might actually happen. If it has been trained on the entire corpus of medical science, and it is indeed intelligent, then surely it can reference ballistics studies and give you a very detailed and thorough theory of what--exactly--injuries you might expect from a 12oz can being dropped from the height of a skyscraper. Calculating the terminal velocity and therefore the momentum of the can is trivial. Characterizing the physics of the impact on various parts of a human body is trivial. If it actually understood your request why didn't it just answer the question?
It's a rhetorical question. LLMs do not "understand". It is completely outside their capability. "Understanding" is something we impose upon their output (to loosely quote TFA). [edit] I think the most powerful evidence for a lack of any understanding whatsoever is all the stuff about the cherries being in or out of the can. Yes, cans contain things. That is not a profound observation, nor is it at all relevant to the question. If you drop an empty can off a skyscraper nothing meaningful will happen. And, no, probably dumping all the cherries out won't hurt anyone or cause a slipping hazard... It's also not particularly relevant to point out that dropping things off skyscrapers is bad behavior. But that's more forgivable from a CYA standpoint.
I believe you are projecting something that is not there onto a completely mindless stochastic process.
[another edit] I found this helpful, you may also: https://www.0xkato.xyz/how-llms-actually-work/
I'll have to think more about this. Your last point feels somewhat persuasive to me, I'm at least a lot less confident in this than I was.
This is a facile point. Lisp expert systems transparently don't understand the meaning of any symbols they process, yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability. The fact that LLMs are less transparent than Lisp expert systems (and easier to program) is extremely bad evidence that they understand language. Especially given that AFAICT Opus does not properly understand concepts like "four."
> yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability
Where can I access such a Lisp expert system?
If I cannot because they don't exist: then they cannot do the same things an LLM can do. And of course one can assert anything and everything about what a non-existing thing could do.
After having used LLMs for some time now, I don't agree with the concept they are just token generators, unless you think that's all humans are too. The way we test in most schools is just picking the right token. We also give them unique problems that they never saw in their training, which is the nature of programming. I realize they are probabilistic token generator models, but I find it harder and harder to accept that somehow there isn't something more going on. I'm not sure whether they are intelligent or not, but for the most part token generation is how you get degrees too. The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
They are just token generators. It is just that 'just' does a lot of lifting!
Here's the thing: most things people do does not involve tokens of any kind. It is, in fact, stuff that not easily describable. For example, it's trivial for a person to walk, but they cannot verbally describe what muscles they're activating in what order to make that happen.
Cognitive skills such as tool use and complex navigation predate language as well. That means there's a core of reasoning in humans that doesn't depend on "tokens" or "language" of any kind. Language is a tool for communication and forming complex human societies, but it's not cognition.
> The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
Well a parrot does perform complex reasoning on novel situations all the time. It just doesn't have the wiring to connect that to "tokenized" human language. I suspect LLMs have the opposite problem, where they exist in the domain of their "tokens" and have no way to connect these to truly novel situations that have no existing words to describe them.
I think this is the most measured take I've seen from Bender, and I think it summarizes her only compelling point well (technologies should be referred to specifically rather than generally as AI, and that referring to everything as AI is not useful and helps hype the technology in a way that benefits those selling it).
In her previous interviews, I've found her assertion that LLMs aren't useful and will never be good at anything totally uncompelling. Also laughed at this quote as she's been pretty harsh IMO on "the people who like the systems".
> it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.