October 19, 2020

The Pendulum Swings Eternal

I shouldn't have had a newsletter this week because of the TLA+ workshop, but I forgot to say "no newsletter this week" last week so guess what, extra newsletter!

Alloy Stream

A couple weeks ago I paired with my friend Chelsea Troy to help her improve an Alloy model. The recording of the stream is now available here! I demonstrate a bunch of useful Alloy refactorings while also balancing a ball on my head. Check it out!

(I recommend skipping to minute 9 to get past some connection issues we had)

The Pendulum Swings Eternal

When you begin studying software history, you notice a common pattern:

• People have a flawed "thing", call it X.
• In order to escape the flaws of X, they turn to Y. Y is flawed too, but it's flawed in different ways. These people compare the flaws of X to the flaws of Y and decide that Y is more appropriate for their use case.
• Use of Y spreads. Over time, fewer and fewer people use X and more people are use Y.
• Eventually, most people are only familiar with Y. They don't remember X and don't remember why people chose Y the first place.
• People use Y. Y is flawed.
• In order to escape the flaws of Y, they turned to X'. X' is flawed too, but it's flawed in different ways...

I visualize it as a giant Foucault's pendulum, swinging back and forth. It's not always the same each time, and often X' is either an improvement on X or a variation in the different part of the X possibility-space. What's more important is the opposition of the two paradigms, X-space and Y-space, so adopting one rejects the other. Some examples:

• Client vs Server: Mainframes, mini- and micro-computers, computation on webservers, computation on webclients. Rendering HTML on the server, then doing it as part of the JavaScript, then back to the server again.

• Up-front design: The ad-hoc ways we wrote software inspiring waterfall, then the method wars, culminating in things like the Rational Unified Process, rejected by Extreme Programming and then Agile. The pendulum is just beginning to swing the other way as things like formal methods grow in popularity.

• Third-party libraries: a lot of people complain about NPM package bloat and dependency chain attacks. 30 years ago, "lack of libraries [was] the reason for software bloat."

Trendy languages going from high-performance to "developer happiness" back to performance again. Cloud vs bare-metal. Monoliths to microservices and back again. Flux and WndProc. Most pendulums are only going to affect a few communities, and many times the swing won't complete before we start reversing course. Like most of the SQL vs NoSQL swings aren't actually that big in the grand scheme of things, but they still follow the same arcs.¹

Why does this happen?

A few reasons. The first shift away from X often comes from a cost-benefit analysis. They know the pros and cons of X and have some suspicion of the pros and cons of Y. For their purposes, it makes more sense to use Y instead of X. But as Y displaces X in the mainstream, two things happen:

1. People use Y by default, even in places where it's the worst choice. This is especially common with newcomers who never even learned about X, just grew up in a context were all there was was Y.
2. People without the mediating counterforces of X's pros and Y's cons become more extreme in their use of Y, moving to places in the design space where things get "weird".² People moving so far from upfront design they reject estimation and planning entirely. People importing package chains for one-liners. Single page apps for serving text.

This creates two counterforces: people looking for an alternative to Y and people skeptical of Y's most extreme forms. Then there are a few things that can draw people back to the X-space:

• Communities using X never actually died out, they just became obscure. They continue to explore the design space and find X', which is more palatable to people. AI hype often follows this, though it's more of a boom-bust cycle than a pendulum,³ as do trends in programming languages.
• X and Y are obvious enough contrast that anybody moving away from Y can rediscover the essentials of X from first principles. Third-party libraries are an obvious example here.
• The contexts of computing change in a way that makes the pros of X much more important. The client versus server divide is exactly this: requirements changing as different types of computing devices go mainstream.

I'm most fascinated by how this always seems to look like "starting over". The people adopting X' usually don't know about X. For all they know, they're adopting the Y-alternative for the first time. The history and context is lost.⁴

(If anything, it seems like a historical perspective can be a liability here. If people do know about X it's usually used to attack X', given that we once had X and switched to Y.)

Why does this matter?

First of all, I think this is basically impossible to stop. If I managed to make upfront design mainstream, in 10-15 years people would start rejecting it again. So there's an element of humility to knowing this. It also means that any paradigms I reject, I'm not necessarily rejecting because they're intrinsically bad, but because the pendulum of popular opinion is moving away from them. Maybe eventually I'll run into circumstances where it's the right choice, or stumble into the variation that makes it much more appealing to me.

More pragmatically, it's a research heuristic. We can look at previous swings the pendulum and get more insight into where we're going by comparing X' to the "forgotten" X. And if all we have is Y and it's not helping us in our current situation, we can use the pendulum to infer the existence of an X and start investigating it as an alternative.

Most of all, though, it's just a useful lens for understanding things. I find I can better make sense of software history and culture when I look at them though this lens.

If you're reading this on the web, you can subscribe here. Updates are once a week. My main website is here.

My new book, Logic for Programmers, is now in early access! Get it here.