Aether Mug - Famous Cognitive Psychology Experiments that Failed to Replicate
A quick reference
TL;DR is the part in bold below.
The field of psychology had a big crisis in the 2010s, when many widely accepted results turned out to be much less solid than previously thought. It's called the replication crisis, because labs around the world tried and failed to replicate, in new experiments, previous results published by their original "discoverers". In other words, many reported psychological effects were either non-existent—artifacts of the experimenter's flawed setup—or so much weaker than originally claimed that they lost most of their intellectual sparkle.
(The crisis spanned other fields as well, but I mostly care about psychology here, especially the cognitive kind.)
This is very old news, and I've been vaguely aware of several of the biggest disgraced results for years, but I keep on forgetting which are (still probably) real and which aren't. This is not good. Most results in the field do actually replicate and are robust[citation needed], so it would be a pity to lose confidence in the whole field just because of a few bad apples.
This post is a compact reference list of the most (in)famous cognitive science results that failed to replicate and should, for the time being, be considered false. The only goal is to offset the trust-undermining effects of my poor memory—and perhaps yours, too?—with a bookmarkable page.
This can't be a comprehensive list: if a study is not on this page, it's not guaranteed to be fully replicated. Still, this should cover most of the high-profile debunked theories that laypeople like me may have heard of.
Credit: I enlisted the help of Kimi K2, o3, and Sonnet 4 to gather and fact-check this list. I also checked, pruned, and de-hallucinated all the results.
Ego Depletion Effect
Claimed result: We have a "willpower battery" that gradually depletes during the day as we exercise self-control. (I remember reading Baumeister's pop-science book and being awed by the implications of their findings; I might have known it sounded too good to be true.)
Representative paper: Baumeister et al. 1998
Replication status: did not replicate
Source: Hagger et (63!) al. 2016
Power Posing Effect
Claimed result: Adopting expansive body postures for 2 minutes (like standing with hands on hips or arms raised) increases testosterone, decreases cortisol, and makes people feel more powerful and take more risks.
Representative paper: Carney, Cuddy, & Yap (2010)
Replication status: did not replicate
Source: Ranehill et al. (2015)
Social Priming: Elderly Words Effect
Claimed result: People walk more slowly after being exposed to words related to elderly stereotypes.
Representative paper: Bargh, Chen, & Burrows (1996)
Replication status: did not replicate
Source: Doyen et al. (2012) (I like how they prove that the psychological effect was actually in the experimenters, rather than the subjects!)
Money Priming Effect
Claimed result: Simply thinking about money makes you more selfish and more likely to endorse free market values.
Representative paper: Vohs, Mead, & Goode (2006)
Replication status: did not replicate
Source: Rohrer, Pashler, & Harris (2015)
ESP Precognition Effect
Claimed result: In some cases, people can predict future events "that could not otherwise be anticipated through any known inferential process".
Representative paper: Bem (2011)
Replication status: did not replicate
Source: Galak et al. (2012), Ritchie, Wiseman, & French (2012)
Cleanliness and Morality Effect
Claimed result: Being clean or thinking about cleanliness makes people more morally lax.
Representative paper: Schnall, Benton, & Harvey (2008)
Replication status: did not replicate
Glucose and Ego Depletion Effect
Claimed result: Connected to the debunked ego-depletion effect, this one claims that adding glucose to your blood "recharges" the willpower battery. (For a while, I may have drunk more orange juice than usual after reading Baumeister's book. At least it's healthy-ish.)
Representative paper: Gailliot & Baumeister (2007)
Replication status: did not replicate
Source: Lange & Eggert (2014)
Hunger and Risk-Taking Effect
Claimed result: People exposed to the scent of freshly baked cookies become less sensitive to risk and take more risks to obtain food.
Representative paper: Ditto et al. 2006
Replication status: did not replicate
Psychological Distance & Construal Level Theory
Claimed result: "Psychologically distant" events are processed more abstractly, while "psychologically near" events are processed more concretely. E.g., you worry about the difficulty of a task if you have to do it tomorrow, but you see the same task's attractive side if it is planned far in the future.
Representative paper: Trope & Liberman (2010), building on Liberman & Trope (1998)
Replication status: serious credibility problems
Source: A collaboration between 73 labs around the world is vetting this theory right now because of many doubts about its validity.
Ovulation & Mate Preferences Effect
Claimed result: Women are more attracted to hot guys during high-fertility days of their cycles.
Representative paper: Gildersleeve, Haselton, & Fales (2014)
Replication status: did not replicate
Source: Stern, Gerlach, & Penke (2020)
Marshmallow Test & Long-Term Success Effect
Claimed result: Children's ability to resist eating a marshmallow when left alone in a room at age 4-5 strongly predicts adolescent achievement, with those who waited longer showing better life outcomes.
Representative paper: Shoda, Mischel, & Peake (1990)
Replication status: did not replicate significantly
Source: Watts, Duncan, & Quan (2018)
Stereotype Threat (Women's Math Performance) Effect
Claimed result: Women risk being judged by the negative stereotype that women have weaker math ability, and this apprehension disrupts their math performance on difficult tests.
Representative paper: Spencer, Steele, & Quinn (1999)
Replication status: did not replicate
Source: Flore & Wicherts (2015)
Smile to Feel Better Effect
Claimed result: Holding a pen in your teeth (forcing a smile-like expression) makes you rate cartoons as funnier compared to holding a pen with your lips (preventing smiling). More broadly, facial expressions can influence emotional experiences: "fake it till you make it."
Representative paper: Strack, Martin, & Stepper (1988)
Replication status: did not replicate
Source: Wagenmakers et (54!) al. (2016)
Objective Measurement of Biases
Claimed result: You can predict if someone is racist by how quickly they answer certain trick questions.
Representative paper: Greenwald, McGhee, & Schwartz (1998)
Replication status: mixed evidence with small effects
Source: Oswald et al. (2013) shows that the prediction power is small at best.
Mozart Effect
Claimed result: Listening to Mozart temporarily makes you smarter.
Representative paper: Rauscher, Shaw, & Ky (1993)
Replication status: did not replicate
Source: Pietschnig, Voracek, & Formann (2010) (What a title!)
Growth Mindset Interventions
Claimed result: Teaching students that intelligence is malleable (not fixed) dramatically improves academic performance.
Representative paper: Dweck, & Leggett (1988)
Replication status: mixed results - many failed replications but also some successful replications
Failed replication source: Li & Bates 2019
Notable successful replication: Yeager et al. 2019 in Nature
Bilinguals Are Smarter
Claimed result: Being bilingual provides substantial cognitive advantages in attention, task-switching, and executive control.
Representative paper: Bialystok, Craik, & Luk (2012)
Replication status: did not replicate
Source: Lehtonen et al. 2018
Did I miss any famous debunked studies? Let me know by replying to this newsletter, and I'll add it to the list. ●
Cover image: Photo by Rebecca Freeman, Unsplash