Software Artifacts and Programming vs Engineering
Hiya everyone! Workshop is all done and I'm alive again. For those of you just joining us, I teach formal methods workshops to companies. Working on a piece about my pedagogy now, actually; for such a large industry there's surprisingly little shop-talk out there.
-
Okay so the last two newsletters were about research and the workshop was all formal methods, so let's take a break from "stuff that pays the bills" and chat software philosophy for a bit. I recently read Improving end-to-end tests reliability. In order to make better E2E tests, they recommend building a lot of infrastructure to support the tests: high-level interfaces, automatic bisection, stuff like that.
This reminded me of a concept I've mentioned a few times before: software as an artifact. One of the fundamental differences between "programming" and "software engineering" is that engineering treats the software as an entity distinct from the thing it's doing— as something that exists in its own right. The software has presence and needs to be supported as much as the client domain.
This is easier to see with an example. We have a program that runs a science simulation. Maybe something like this. Give it a quick skim. Now, think of some ways you can improve that code.
...
Got some? I'd be surprised if you didn't. I'd also be surprised, though, if any of your changes actually solve the problem better. In the context of this code, the only thing that matters is actually getting the data! Do your changes lead to faster runtimes or more accurate results? I'd bet not.
That's because you're looking at this with the eyes of a software engineer, not a scientist. You're more likely to suggest improvements which do completely unrelated things, like:
- Allow use of several different configurations on one copy of the code
- Introspect the program to figure out what it's doing where and when
- Make changes with a lower risk of breaking the program
- Catch a change that does break the program
- Change the algorithm and still be able to reproduce old runs and results
- Share the code with other people
To do any of these things, you need to stop thinking about the software as the code that solves your problem" and start thinking of it as a distinct artifact. You need to treat the code as its own domain and then write code to solve that domain logic. Software engineers implicitly understand this. Scientists don't, which is why science code is such a train wreck.
Code Tiers
But what about the code that supports the code? Is that an artifact too? Let's go back to the E2E article. Conventional wisdom is to keep your tests as simple as possible, because you'd have to test your tests! This suggests to me that we distinguish between the business logic (must be engineered) and the test logic (can be programmed). You have two tiers of code: the domain logic is an artifact and must be supported, but the code that supports it isn't also an artifact.
If you follow the article's recommendations, you instead have three tiers of code: the domain logic is an artifact, the test code is also an artifact, which must be supported by test test code. The test² code, presumably, isn't an artifact. Which makes sense to me. If you had to engineer the test² code, then you'd need to support it by test³ code. Should that, too, be engineered? You have to stop somewhere.
It is interesting to me that we test² code to support end-to-end tests. I don't think unit tests need to be supported in the same way. Extending this idea, I'd predict that property-based testing needs to be engineered, too. To really make PBT work we need to write a bunch of complex abstractions and then test those abstractions. Maybe you'd need four tiers of code if you support the PBT with E2E tests or the E2E with PBT.
What also follows this two-tier model? Deployment scripts— the "infrastructure as code" movement is arguably all about fixing this. Logging and profiling, maybe? Personal use tools. Anything where you have "scripts" in a project. Probably a ton more that I'm not thinking of right now.
Anyway, I think there are other differences between "programming" and "software engineering", but the "software as artifact" is arguably the biggest.
Dunning-Kruger don't real
Not really related to Computer Things but I haven't been able to talk about anything else for two days so I'm dumping it on you nerds too: I recently discovered there might not really be a Dunning-Kruger effect. I tweeted about it here:
The DK paper grouped test scores by quartile and plotted scores and self-perception, giving the famous X-shape. Turns out, you get the same shape if you do the same with completely random data! What you need to do is compare each individual's test score and self-assessment. pic.twitter.com/BzBSR3RDVj
— Inactive; Bluesky is @hillelwayne(dot)com (@hillelogram) January 23, 2022
(Link is to second tweet in the storm, not the first, because Buttondown is doing something weird where they're putting the entire linked article in the tweet embed ¯\_(ツ)_/¯)
If you're reading this on the web, you can subscribe here. Updates are once a week. My main website is here.
My new book, Logic for Programmers, is now in early access! Get it here.