"Integration tests" are just vibes
New blog post, and some thoughts on what makes an integration test an "integration" test.
New blog post! Software Friction is about how all the small issues we run into developing software cause us to miss deadlines, and how we can address some of them. Patreon here.
"Integration tests" are just vibes
You should write more unit tests than integration tests. You should write more integration tests than unit tests. "Integrated tests" are a scam.
But what exactly is an integration test? Is it tests that cross module boundaries? Is it tests that depend on several pieces of behavior, even in one module? Is it anything that tests "emergence"1? Anything with side effects? If get_active_users() runs a database query, are all tests against it integration tests?
No wonder both Twitter and Google have (at one point) claimed that "integration test" wasn't a useful term. There are just too many different things people mean "integration".2
Even if we can't define integration test, we still have recognize them. There are tests that are bigger than our "unit tests" and smaller than an end-to-end test, and people have strong opinions about writing more or less of them. We can have a sense of something being an "integration test" even if we can't strictly define it.
I think it's more useful to think of integration tests in terms of family resemblances. There are certain measurable properties of tests, and integration tests measure relatively different than a unit test. One quality is test speed: unit tests are "fast", integration tests are "slow". This does not mean all integration tests are slower than any unit test (unit property tests can be very slow). But slow tests are more likely to be integration tests than unit tests.
"Integration tests" are a vibe we get based on how we perceive a test's properties. Some of these properties are:
-
The number of lines of production code executed as part of the test.
-
Number of boundaries crossed, module/object or otherwise.
-
Number of unit tests you can decompose the test into.
-
Number of external services required to run the test.
-
Amount of engineering effort to write/run the tests. Does it need separate files, a separate language, setting up lots of docker instances?
-
Nonlocality of discovered bugs: if the test fails, how many lines you have to look at to find the problem.
-
Likelihood a test failure is due to "emergence": things are correct in isolation but interact in a problematic way.
The more of these have high values, the more likely a test is to be an integration test.
Miscellaneous Thoughts
A test that hits the database (external service) can still be a unit test if it runs quickly and covers a small amount of code. Tests that make HTTP calls are less likely to be fast, so more likely to be integration tests.
Mocks and pact-style testing push some of these values down, by reducing boundaries crossed and lines of code executed at the cost of increased test infrastructure. Contracts and assertions increase bug locality, since the test fails the moment a pre/postcondition is broken instead of when the test completes.
If you push the values high enough, do you get end-to-end tests instead of integration tests? Or are E2E tests fundamentally different from integration tests in some way?
Is it possible for a unit test to uncover emergence, or is emergence a sufficient condition to say that something is definitely an integration test?
Things I'm curious about
Gonna make this a weekly thing maybe (or whenever I have enough new stuff to ask about).
- Sokoban. Sokoban is "easier" than Chess or GO (PSPACE vs EXPTIME), but Sokoban AIs regularly fail at simple puzzles that humans can do (ref ex1 ex2). What makes Sokoban "so much harder" for AIs? I don't think it's just a lack of interest because I've found Google and Deepmind papers on solving sokoban (and not making much progress).
- NFCs: I impulse-bought a bunch of NFC chips. I'm already using them to open train trackers, what else can I do with them? I'm not a mobile dev but can whip up Taskers and server code pretty quickly.
If you know something about either of these, feel free to share! Or write about them publicly and share the link with me. You can just reply directly to this email.
If you're reading this on the web, you can subscribe here. Updates are once a week. My main website is here.
My new book, Logic for Programmers, is now in early access! Get it here.