Grind Smarter, not Harder

New Essay: Why Specifications Don't Compose

Read it here! Didn't sneak peek it to the newsletter because I used a lot of Hugo preprocessing, so this is the first time y'all are seeing it. Enjoy!

May TLA+ Workshop

Just one slot left! May 24-26. Learn how to find bugs that would slip right past types, test, and code review.

Grind Smarter, not Harder

Was gonna post this last week but the workshop took all of the wind out of me. Anyway! There's this article I really liked from Jacob Kaplan-Moss, called Embrace the Grind. The thesis is "software people look for automation solutions to their problems, but a lot of the most important things we can do involve lots of tedious work, and we should embrace that."

I often have people newer to the tech industry ask me for secrets to success. There aren’t many, really, but this secret — being willing to do something so terrifically tedious that it appears to be magic — works in tech too.

In principle, I agree with this. Hell, that's how I first became "known" in wider software: being the person who would spend months slogging through writing documentation. Grind sucks, but it's often necessary, and we can get a lot further than if we try to avoid it entirely.

But there's one really important caveat to that: "grind" vs "automate" isn't a binary, it's a spectrum. While we can't automate our problems away, we can use automation to reduce the grind

When I was writing Why Do Interviewers Ask Linked List Questions?, I wanted to find all uses of "pointer" in 80's era Usenet posts. No problem, that's just a grep, which got me ~2000 Usenet posts. But some posts used it as a technical term, while others used it as a regular English word. That's where the grind comes in. I had to manually check all 2000 posts and filter for only the ones that used it as a technical word.

Before I started, I added some Vim commands.¹ ,n loaded the next untagged message into a buffer and highlighted all uses of pointer. ,g tagged the post as "good" (uses technical pointer) and loaded the next one, while ,b tagged it as "bad". Finally, because people are fallible, ,u loaded back the previous message and untagged it, so I could go over it again. There was still a bunch of grind, as I had to read the posts and check their usage, but the manual effort reduced to typing ,g,g,b,g,b,b,u,g … That made the task faster and painless.

The converse of this is also true: there are hard automation tasks that can be made much easier if you're willing to accept a little bit of grind. My favorite example is a lot older, the first time I was seriously using programming for work. It was when I still told myself I wanted to be a physicist and was interning in a crystallography lab. I needed to split the double peaks of a separate the energy peaks of a diffraction curve. MATLAB has good curve fitting software which makes this part easy, once it found the curves. We had noisy data, meaning the curve fitter would often grab some of the noise and calculate nonsense values from that.

After a week of trying to smooth things out, and then another week of trying to improve the curve-fitting, I finally found the perfect solution:

1. Do all the necessary preprocessing.
2. Display the diffraction graph.
3. Hit a breakpoint in the script.
4. While in debugger mode, look at the graph and manually pick start and end points for the curve-fitting function that only covered the double-peaks.
5. Resume the script.

This took a little more effort than having a fully automated solution, and it had lower analysis throughput, since a human had to get involved. But it did unblock the project, so everybody was really happy with the work. Adding a little bit of grind made the automation possible.

Interactive Automation

This is a different style of program than most of the stuff we write. It's not automated, but it's also not "fully interactive"—it accepts user input but doesn't abstract away the "code" part of the program from the user. "Normal" interactive programs are handcuffed by that design towards complete nonprogrammers. Whereas this style is more like

1. Run some code
2. Do some manual work
3. Tweak the code if necessary
4. Run more code
5. Etc

And this requires different affordances than regular automation needs. For one, better user input, which is normally the domain of end-user applications. That's something that most programming languages don't emphasize. Sure, Python has input, but how do you open up a file picker?

import tkinter as tk
from tkinter import filedialog

root = tk.Tk()
root.withdraw()

out = filedialog.askopenfilename()

By contrast, here's how you do it in AutoHotKey:

FileSelectFile, out

In one case, the file picker is buried in a module of a module and causes all sorts of bug complaints. In the other, it's a top level construct. Very different affordances for the same thing! This matters because the harder it is to make things interactive, the less likely you are to eliminate grind via programming.

(This all reminds me a lot of my idea for an antipattern language. Maybe this is the main use case of an APSL? Doubly interesting because many people suggested shell scripting as best APSL, and shell scripting is… not great for highly interactive automation.)

Then again, there's nothing forcing us to do all interative automation (IA) with the same programming language. A lot of my grind-toolchains are a mix of Python/Powershell and VimScript/AHK. One language interacts and one automates.

We can also think about what makes a good IA scripts. One I've consistently found is to minimize the amount of input necessary from the human, even to individual keystrokes. Another is to have an "escape hatch" to manually do things that aren't covered by the automation. A third is to make mistakes undoable. As an example of these properties, imagine we need to move two hundred files into three folders, and we need to open each file to know where it's supposed to go. Here's where I'd start with an IA:

1. Create a move.json file, which will map each file to a directory string.
2. Assign two hotkeys, like n and p, to close the current file and open the next/previous file in the directory.
3. Assign three hotkeys, maybe even 1 2 3, to "add this filepath to move.json as a key, with directory[i] as the value." A fourth hotkey opens a file picker to choose a different directory, for special cases.
4. Once I reach the end of the files, go through the JSON and inspect it for errors.
5. Run a separate script that reads the JSON and moves the files to the appropriate directories.

If I was doing fewer files, or expected to make few mistakes, I might skip the move.json and have 1 2 3 directly move the files. If I made any mistakes, I'd note them down and move the file to the appropriate directory manually. That reduce IA development time at the cost of more grind. Either way, though, it'd be a lot less grind than without the scripts.

Once I started regularly writing IA scripts, I noticed all the grind in my computing. Both long, tedious tasks and lots of small things that would wear me down over time. Stuff that was impossible to automate so I didn't even think of them as something to automate. But interactively automating them is a lot more feasible, so I do it more, and it's made my computing experience a lot better.

Can't think of a good ending to this, so I won't. Endings are hard. Enjoy the idea, I guess!

If you're reading this on the web, you can subscribe here. Updates are once a week. My main website is here.

My new book, Logic for Programmers, is now in early access! Get it here.