April 4, 2022

The Esotech Lit Gap

I work with a lot of really esoteric technologies, like TLA+, Alloy, J, MiniZinc, and PRISM. Pretty much every non-mainstream technology has a "literature gap" between beginner and expert work.

Take J. If you go around the internet, you'll find lots of articles extolling the virtue of array-based programming and a lot of experts showcasing the cool things you can do with J mastery.¹ Here's what you won't find:

• The best way to structure your code
• Common troubleshooting issues and how to get around them
• Common antipatterns
• Program dissections
• A well-organized documentation of the standard library with examples²
• Blog posts of people discussing techniques without needing to introduce basic concepts first

In other words, there's material to get you started, and material for experts to share with other experts, but nothing for people to go from knowing the language to being a competent practitioner (CP). I'm using J as an example here, but almost all esoteric technology I know about has the same problem. This points to a user gap, where most of the users are beginners or experts. Experts don't need to talk about this stuff because they already know it, so it's not interesting to them. And beginners don't know enough to talk about it usefully.

Presumably, as beginners progress into CP-hood, they'll start sharing the intermediate stuff with other beginners. But it only makes sense to share if you think there will be an audience. With esotech there's almost no "community"— most beginners learn in isolation and don't know of any other learners. The disincentivizes putting in the hard work to improve the literature gap.

It's worth providing a couple of contraexamples— communities where there isn't a lit gap and why one doesn't exist, in comparison to standard esotech.

1. There's a lot of material out there for mainstream languages, like Python, Go, Java, as well as mainstream techniques, like CI/CD and TDD. This is because these all have huge communities, so there's enough CPs to talk with each other, and it's clear there will always be enough beginners to read your stuff.
2. Haskell isn't as fringe as the esotech I listed earlier, but it's significantly more fringe than mainstream languages. But there's a lot of CP material, which I'm quite jealous of. I suspect it's because Haskell has a strong sense of "culture": if you're learning Haskell, you will quickly find other Haskell enthusiasts. This lets the community punch well above its weight level in closing the lit gap.

The lit gap is a serious problem for an esotech. Without literature, it's very expensive for a beginner to become a CP. Most remain give up at, or remain stuck in, the beginner stage, which means fewer people contributing to the literature, the tooling ecosystem, doing case studies, etc. A lit gap perpetuates the structural problems that perpetuates the lit gap.

Closing the Lit Gap

Lit gaps don't close naturally; someone needs to intentionally invest the energy to close it. Fortunately, it doesn't take too much energy to get the process started, and once it's started it gets easier to work on it over time.

I managed to close some of the lit gap in TLA+. Between my books, blog posts, and talks, I've significantly increased people's interest in TLA+, and that's lead to more material being created. I don't think I'm an exceptional teacher of things, I've just put more time than most into getting good at it. IMO anybody else can get similar results with their esotech of choice,³ though I think you want three or four people to really make it work.

(There's still a lot of work to do in closing the TLA+ lit gap. That's why I'm currently revamping learntla. But it's still clear to me that even one sufficiently motivated person can do a lot to make a language more popular.)

Sidenote: gap identification processes

There's one particular part of the lit gap that fascinates me: information that's available but not catalogued.⁴ That's where you can find the information if you look really hard, but it should be easier to find.

To pick on J again, there's a data type called the box which is absolutely essential to doing anything really complex in the language. But in the official site, information on boxes is scattered across fifteen primitives and pages. The boxing function barely explains anything, nor does the glossary, so you're left thinking it's just a way to bypass the homogenous array requirement. You only realize how important boxes are when you've been elbow-deep in the language and noticing all of the different places they crop up.

That should all be in one page. That would remove 1) the frustration of learning it by experience and 2) the investment required to learn it by research.

Cataloging represents an interesting case where the gap is automatically identifiable. A technical writer could sit down with an expert and go through a set of routine questions, and by the end of it have a list of things that need to be better catalogued. I wonder if there are other classes of information where you can do the same. Maybe "gap identification processes" is itself a useful discipline?

If you're reading this on the web, you can subscribe here. Updates are once a week. My main website is here.

My new book, Logic for Programmers, is now in early access! Get it here.