Social constructions and high rates of scientific false positives
We take a brief look at how something seemingly-technical like "selling out" an event is a social construction, and we reason from basic principles how much of published science might be false positives.
I hope your week is going well!
New articles
Quarterly Cup 2024 Q4 Retrospective
This past quarterly cup went okay-ish. I really wish I could spend more time forecasting, because it's one of those things it's really fun to do well on. Perhaps the most interesting takeaway from this is how even seemingly-technical concepts are actually socially constructed.
Full article (1–2 minute read): Quarterly Cup 2024 Q4 Retrospective
Flashcard of the week
If 10 % of the hypotheses tested by the scientific community are actually true, and the average power of their studies is 30 %, and the average p value ends up being 10 % in practice, and only positive results are published, what fraction of published results are actually true?
I find it easiest to reason about this by starting with a hypothetical set of 100 experiments, rather than trying to reason directly about the probabilities.
Of 100 experiments, 90 test hypotheses that are false, and 9 out of these will by accident get a significant result. There are 10 true hypotheses in there, of which only 3 are successfully shown to be true. This means 25 % of the published results (3:9) are actually true, the other 75 % are false positives.
I suppose this counts as "food for Fermi estimation" even though it depends on rather specific assumptions. It doesn't seem entirely unreasonable to me that 75 % of published results are false positives. Good to keep in mind!
Premium newsletter
Once I've forecast on all questions in the ACX 2025 competition, I will send out the next premium newsletter. It will contain three great links, two book recommendations (covering investing and polar expeditions), a link to a future article that's not yet published, and the ACX 2025 forecast rationales.
If any of this sounds interesting, you should upgrade to a premium subscription! If you upgrade now, you will pay only $2/month – as interest increases and I learn to hit my stride with publishing these, the price will be set higher for future subscribers.
As always, you can cancel at any time. To upgrade, click the subscription link at the top of this newsletter and fill in your email again.
If you subscribe now, you should also get access to the previous premium newsletter which went out a couple of weeks ago. It contained three great links, and nearly three full-length articles, on topics as diverse as:
- How to talk to children (and uncooperative adults),
- How insurance companies buy insurance, and
- How I made my forecasts in the 2025 Vox Future Perfect forecasting competition.
Your opinions
I cannot improve without feedback. Reply to this email to share your thoughts on any of the topics above, or anything else!