New Essay! (Also Workshop Field Notes)

Workshop Improvements

Apologies for not sending anything on Monday; I was doing a workshop for Facebook! I had a blast, but I'm usually dead by the end of each day, so no time to write long rants about computer things.

As an instructor I'm always looking for ways to improve my workshops. Unlike with programming or even writing, there's no tight feedback loop I can use to actually check my ideas. If I want to see if an idea works, I have to implement it and try it out at the next workshop. So here's some of the new stuff I tried that worked:

Group Exercises

My workshop used to be split into labs and exercises. Labs were specs we all studied and modified as a group, exercises people did individually. There are big gaps in performance on the exercises, and often things that seem simple to me take a long time for beginners. There were two problems with my exercises. First, they were about applying skills, not developing them. That means that the student needs to do more and synthesize more ideas for each exercise. And with that many moving parts, there were many things someone could get stuck on. If you need to do four things, where A, B, D all take a minute, but C takes 10 minutes due to you getting stuck on it, then it takes you 13 minutes. If the person next to you breezes through C but got stuck on B, they would also take 13 minutes. But if you worked together, it'd take you around 4 minutes.

I remade most of the exercises as group excercises, where the whole class does them together, out loud, with me listening and making corrections. People completed these much faster and felt like they got a better sense of what was going on this way.

Individual Exercises

Group exercises are complicated, real world, interesting. I made a new set of individual exercises that were simple, abstract, and boring. One group exercise is about modeling invariants for high-redundancy data replication. One individual exercise is "write a spec that counts from 1 to 10." People complete them very fast. But this saves a lot of time going forward, as now people have practiced the low-level skills. You won't get stuck setting up your PlusCal spec if you've already set up 10 other PlusCal specs, even if each individual spec was trivial.

I didn't have enough individual exercises, and some were still on the "apply" side, but the ones I did have helped a lot. Adding more individual exercises will probably be my next big expansion to the workshop.

Graduated Cheatsheets

This was the one I was most excited about.

A good cheatsheet is a single page reference that has all of the syntax and semantics on it. Instead of looking through all the material for function syntax, you can look on the cheatsheet. I wrote a TLA+ cheatsheet earlier this year and it's helped people so far. But in the past couple of workshops, I noticed a fundamental problem: a cheatsheet has all the information, but earlier most of it is noise! For example, if the workshop covered topics A-D, and the reader has only learned A and B, then C and D are just clutter. Worse, they might not realize that C and D are topics they don't know yet, so will try to infer and apply them. They're more likely to try the wrong thing, and have a harder time finding the right thing.

Instead, I could create four cheat sheets. The first has A, the second has A and B, etc. I pass out the new sheet once we've covered the material and have started exercises. So at any given point the student's cheatsheet has only and exactly the material they know.

We can do better than that, though! The cheatsheet is normally group by logical unit, not learning unit. Imagine if D was a more advanced version of A, while B was a prerequesite to C. Then I'd want to lay out the cheatsheet like this:

A D A
B C C

(If you're not quite following, imagine that A is "loop syntax", while D is break and continue. Those keywords should be next to loops, but I might cover them later.)

The second cheatsheet should have only A and B. But I decided to preserve the final layout:

A   A
B    

My theory was that this would give people a geometric familiarity. If they looked up A on the old cheatsheet, when they got to the final one, they'd "instinctively" know where to look for A. They won't spend time parsing C and D, because they know that's not where A is supposed to be.

In practice, all my ideas worked out. People who used the earlier cheatsheets consistently quickly found the topics they needed, while people who didn't use them spent more time parsing irrelevant information. I have a lot of tweaks to make, but I really like where this is going.

And here's a few things that didn't work:

Student-Led Exercises

There are two ways to do a group exercise:

1. I lead, they have to tell me exactly what to type
2. Someone else leads, we walk them through it.

I thought (2) would be more useful to people. In practice, people stopped paying attention when I did this. They started doing the exercises individually, assuming that it was more of me helping one person and not the whole class. When I lead, they actively engaged. Going forward, I need to lead all group exercises.

Specific Labs

I had a couple of group labs that focused on practicing specific skills, like writing invariants. Unfortunately, these required too much conceptual setup time: I can't talk about invariants of blue-green deploys if I have to spend fifteen minutes explaining how blue-green deploys work! They won't have the intuition required to see useful invariants. I still like the idea of talking about DevOps as a lab, but it would be more "let's practice writing a spec" versus "let's improve our skills with invariants."

Cheatsheet Troubleshooting

I added a new second to the cheatsheet called "troubleshooting", to help them fix common issues with their specs. That'd be things like using |-> when they meant ->, quotes in structs, things like that. In practice, people didn't look at them. Next time I will move them into the class materials, and then walk people through at the start so they know to check their for initial issues they have.

So how'd it all work out? Overall, I estimate the changes put us half a day ahead of schedule. For a 3-day workshop, that's a very, very big deal. It meant an extra half-day to actually apply TLA+ to their actual work problems. And remember, this was after adding a bunch of new material! There's still a lot I can do to improve the cheatsheets and exercises. I'm hopeful that after I'm done optimizing, it'll speed things up by another half day. Then I can add in more advanced topics!

I also need to apply all these improvements to my Alloy workshop. With that I might even be able to offer a barebones one-day workshop, as opposed to the minimum two that Alloy takes.

Increment Magazine

I've been published in the Increment Teams Issue! My piece is The epistemology of software quality and hands-down wins for "best hero art in the issue", an award I played no part in getting. I had a blast working with the Increment folks and 100% recommend reaching out if you want to write for them.¹ Check it out!

¹ Funny story here: I was approached by them to write for the previous issue on testing. Natch the piece was about formal methods and the limits of unit testing. Unfortunately the issue ran into space constraints and I volunteered to pull the piece. So now I have a 3,000-word final draft of "The Limits of Testing" sitting on my hard drive. I should probably put it on my blog or something

If you're reading this on the web, you can subscribe here. Updates are once a week. My main website is here.

My new book, Logic for Programmers, is now in early access! Get it here.