Check if your project is impossible
Avoiding toxic optimism in your data projects
Welcome to Data Dash
Compressing an avalanche of thoughts about data into byte-sized chunks. In your inbox every two weeks on Wednesdays.
Throw the rock down the hole
Data work is unpredictable in ways software engineering isn’t. A software engineer knows it’s possible to build a landing page. A data person can’t know ahead of time if their model will show anything useful.
However, this non-deterministic nature can lead to toxic optimism about data projects. If nothing can be determined for sure ahead of time, isn’t anything possible?
This kind of attitude can lead to doing the equivalent of not tossing a rock down a hole before climbing into it. Sure, we won’t know precisely how deep the hole is no matter what. And if we stand there for ten seconds and still haven’t heard the rock hit, the hole is too deep to climb today.
A couple practical examples
I hope there’s zero rock throwing during your data projects, so here’s some actual examples.
If you’re thinking of running an A/B test, simulate the kinds of data you expect to get before launching the test. This will give you a sense if you have a prayer of answering the key research questions. Try not to give into the temptation to inflate the expected effect size. Now go ahead and halve that effect size again.
If you’re trying to find a causal mechanism, do you have a DAG? Can you measure the key elements of that DAG? Do you have the resources to implement the research design necessary to test that DAG? I think the answer to at least one of those questions is almost always no, and admitting that upfront will save you a lot of grief. Do some solid qualitative research instead to get at the “why” of what you’re seeing.
To argue against myself
Sometimes what makes the project “impossible” is lack of knowledge in a certain area. I avoided projects requiring The Terminal before it became one of my favorite tools. There’s nothing wrong with lacking knowledge, and sometimes the best decision is to change scope. I’m a big fan of getting to small wins, and finding those can be a more helpful approach than abandoning the project.
A data thing I liked
Write code with your alphabet radio on
A non-data thing I liked
This stream of the horror game The Backrooms