An article has been circulating that exposes the secret underpinning much of social science theory. Ethan Watters, who previously chronicled the spread of western-style mental illness around the world, now discusses a global flaw in behavioral science research.
February 28, 2013
Writing in Pacific Standard he discusses a paper that appeared in a shortened version in Nature and a longer version in Brain and Behavioral Sciences, pointing out many of social scientists’ assumptions about human nature come from experiments drawing from on a relatively small subset of the world’s population:
A 2008 survey of the top six psychology journals dramatically shows how common that assumption was: more than 96 percent of the subjects tested in psychological studies from 2003 to 2007 were Westerners—with nearly 70 percent from the United States alone. Put another way: 96 percent of human subjects in these studies came from countries that represent only 12 percent of the world’s population.
This bias in sampling is the infamous “college student problem,” and if you ever had to participate in an experiment for credit in a college class, you’re familiar with it (of course, you might have been allowed to opt out and write a paper instead, but do you know anybody who actually did that?) College students are plentiful and convenient to where the researchers usually are, can be encouraged to volunteer, and as the punchline to the old joke goes, “there are some things a rat just won’t do.”
When Joe Henrich, who trained as an anthropologist but now has a job split between an economics and a psychology department, started taking classic economics experiments on the road, he found radically different results in small-scale societies around the world, calling the assumptions of universality into question. Among the indigenous Machiguenga in Peru, he tried the classic ultimatum game, in which two people work to split a sum of money. (One person gets to propose how the money should be divided and the other can only accept or reject the offer. If the second rejects, neither get any money.) Henrich found that the Machiguenga often agreed to splits that American players would reject as “unfair”. In other words, they played the game more “rationally” than American college students, as accepting any deal increases the well-being of both.
But this failure didn’t apply to only economic games, when Heinrich started cataloging the findings which failed to replicate around the world with experimental psychologists, Steven Heine and Ara Norenzayan they concluded, writing in Nature:
Experimental findings from several disciplines indicate considerable variation among human populations in diverse domains, such as visual perception, analytic reasoning, fairness, cooperation, memory and the heritability of IQ. This is in line with what anthropologists have long suggested: that people from Western, educated, industrialized, rich and democratic (WEIRD) societies — and particularly American undergraduates — are some of the most psychologically unusual people on Earth.
So how do you solve this problem of theories developed by WEIRD professors based on data from WEIRD participants failing to generalize outside the Western world? If you’re IPA, you take it to the streets. Streets in urban slums, rural villages and everywhere in between to see how the actual people affected by policies will behave when they’re implemented. We often find unexpected results, programs that sounded good but didn’t work, or that not everybody in our study responded identically. What we’ve learned is that you can’t know if what sounds good in a government or NGO office will work well without a real evaluation and actual people from the communities involved participating.
Within each study, we rely on our local staff to make sure we’re asking the right questions and pilot relentlessly. Our Uganda office spent a full year piloting the super savers program before the main study began.
IPA’s also trying to help other researchers get over their WEIRD issues. Most behavioral researchers can’t find another society, hop on a plane, run some studies there, and be back in time for their 3PM lecture, but if we’re going to create policies that work, it’s important to make sure that economic principles take into account real people. To that end, we’re supporting the Busara Center for Behavioral Economics to help researchers run studies in Nairobi, Kenya. The center has a lab adapted for the needs of behavioral researchers and the local population, which include touchscreen computers designed for non-literate participants to use, and a pool of 1,000 people signed up to participate who are paid for their time by MPesa mobile money transfers (more here).
We like to think it’s important for economic principles to actually work, otherwise, you just end up with WEIRD policies.