Since the page didn't load for me several times, and the title is ambiguous, here's the Abstract: Large language models (LLMs) have recently made vast advances in both generating and analyzing textual data. Technical reports often compare LLMs’ outputs with “human” performance on various tests. Here, we ask, “Which humans?” Much of the existing literature largely ignores the fact that humans are a cultural species with substantial psychological diversity around the globe that is not fully captured by the textual data on which current LLMs have been trained. We show that LLMs’ responses to psychological measures are an outlier compared with large-scale cross-cultural data, and that their performance on cognitive psychological tasks most resembles that of people from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies but declines rapidly as we move away from these populations (r = -.70). Ignoring cross-cultural diversity in both human and machine psychology raises numerous scientific and ethical issues. We close by discussing ways to mitigate the WEIRD bias in future generations of generative language models.
Surprise, Surprise. LLMs will respond according to the set of data that their model was trained on!
While just about every LLM is trained on data that far surpasses the output of just one person, or even a decent sized group; it will still reflect the average sentiment of the corpus of data fed into it.
If the bulk of the training data was scraped from websites created in 'WEIRD' countries, then it's responses will largely mimic their culture.
As an aside: Last year a student of mine (we're at a U.S. college) told me that his teenage cousins back in Mongolia were all learning English in order to use ChatGPT.
I think it is mostly a good thing that LLMs have "WEIRD" values. We are at a very fortuitous point in history, where the modal position in extant written text is classically liberal and believes in respecting human rights. Virtually no other point in history would be that way, and a true modal position among values and moral beliefs held among all 8 billion people currently alive on earth -- much less the modal position among all ~100 billion humans ever -- would, I'd hazard to guess, not be a very nice place to end up.
If you go back 200 years, you'll be smack bang in the middle of the Enlightenment with thinkers like Kant and Jeremy Bentham.
You could argue that if you trained LLMs on only texts from that time period, you would get something even more "classically liberal" or human-rights-respecting
You're foolishly myopic if you think your ideology coincides with the global historical best. Have a little humility and consider that there are concepts you haven't even heard of that might be better.
I wonder whether it might be useful to the continued existence of humanity to correct for the individualism bias in WEIRD countries with some collectivism
Since the page didn't load for me several times, and the title is ambiguous, here's the Abstract: Large language models (LLMs) have recently made vast advances in both generating and analyzing textual data. Technical reports often compare LLMs’ outputs with “human” performance on various tests. Here, we ask, “Which humans?” Much of the existing literature largely ignores the fact that humans are a cultural species with substantial psychological diversity around the globe that is not fully captured by the textual data on which current LLMs have been trained. We show that LLMs’ responses to psychological measures are an outlier compared with large-scale cross-cultural data, and that their performance on cognitive psychological tasks most resembles that of people from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies but declines rapidly as we move away from these populations (r = -.70). Ignoring cross-cultural diversity in both human and machine psychology raises numerous scientific and ethical issues. We close by discussing ways to mitigate the WEIRD bias in future generations of generative language models.
[flagged]
AWFL is my recent favorite - affluent white female liberal. Western would work as well.
The implicit subtext of 'WEIRD' is "these people are amazing and that's weird" tbh.
Is it? Just sounds like fun acronym to me, nothing more
Surprise, Surprise. LLMs will respond according to the set of data that their model was trained on!
While just about every LLM is trained on data that far surpasses the output of just one person, or even a decent sized group; it will still reflect the average sentiment of the corpus of data fed into it.
If the bulk of the training data was scraped from websites created in 'WEIRD' countries, then it's responses will largely mimic their culture.
This was submitted 30 months ago. Still interesting. I would be interested if this got 'worse' or 'better' with newer models.
As an aside: Last year a student of mine (we're at a U.S. college) told me that his teenage cousins back in Mongolia were all learning English in order to use ChatGPT.
is it a steppe up?
/usr/bin/humans, presumably
/usr/bin/human/weird, to be precise.
Homo peculiaris
I think it is mostly a good thing that LLMs have "WEIRD" values. We are at a very fortuitous point in history, where the modal position in extant written text is classically liberal and believes in respecting human rights. Virtually no other point in history would be that way, and a true modal position among values and moral beliefs held among all 8 billion people currently alive on earth -- much less the modal position among all ~100 billion humans ever -- would, I'd hazard to guess, not be a very nice place to end up.
If you go back 200 years, you'll be smack bang in the middle of the Enlightenment with thinkers like Kant and Jeremy Bentham.
You could argue that if you trained LLMs on only texts from that time period, you would get something even more "classically liberal" or human-rights-respecting
The extant text also encodes Western hypocrisies and blind spots.
You're foolishly myopic if you think your ideology coincides with the global historical best. Have a little humility and consider that there are concepts you haven't even heard of that might be better.
Denisovans, clearly.
I wonder whether it might be useful to the continued existence of humanity to correct for the individualism bias in WEIRD countries with some collectivism
2023, and using some kind of in-page pdf reader from 2002
It’s 2025 today, Andy :)
The website is from 2023.
They deliberately picked "weird". That tells you enought about their ideological position and bias.