Why p values should be distrusted and why one should dig deeper than the immediate problem
We briefly work through why the p value specifies the lower bound for the risk of false positives, look at a common – and silly – source of error beyond that, and then rehearse why we shouldn't blindly accept requirements as given.
I hope your week is going well!
New articles
The Lying p Value
We often think of the p value as the risk of false positives, but that's giving it way too much responsibility. It specifies the risk of false positives due to sampling error, assuming the rest of the methodology and analysis is flawless.
In other words, the p value is a lower bound on the risk of false positives. The rate of false positives can be higher than the p value, but never less than it. One trivial reason for higher rates of false positives is computing the significance threshold wrong!
Full article (1–2 minute read): The Lying p Value
Flashcard of the week
Common product development advice is to dig beyond the immediate task one is faced with.
Why should you spend time to understand the problem being solved, rather than just accept requirements?
As with the earlier flashcard on the element of surprise, I have long been able to give a rambling, imprecise answer to this. But there are two words that answer it concisely:
People solutionise.
The person who asked you to help out with something have not given you the actual obstacle they are facing. Instead, they immediately jumped to a solution that seemed intuitive to them, but that might not be good in the grand scheme of things. What they present to you might be a problem relating to a solution for a problem that arose from a solution they are considering to the original problem – or even more steps removed from the original problem.
Going back to the original problem both makes assumptions explicit and allows for more creative and effective problem-solving.
Premium subscription
I'm gauging interest in additional, monthly premium content[1]. For a limited time, a premium subscription to this newsletter will cost just $1 per month to see how large the interest is.
To upgrade, click the subscription link at the top of this newsletter and fill in your email again.
[1]: To be determined more precisely, but ideas include
- links to other articles,
- book reviews,
- previews of future articles,
- voting on topics for me to read/write about, etc.
Your opinions
As always, I cannot improve without feedback. Reply to this email to share your thoughts on any of the topics above, or anything else!