June 22, 2020

25: Append-only notetaking

Today I'd like to talk to you about append-only logs, the simple data-structure behind some of the coolest peer-to-peer systems out there today. Then I'd like to convince you why they're useful for something as simple as a notetaking system (you can probably see where this is going).

First, some housekeeping. This is A Warm Newsletter, a weekly log of the things I (jared) have been working on and thinking about. For the last couple of weeks I've been detailing my work on fancynote, my experiment with making my own notetaking software from scratch.

But instead of actually working on the fancynote task I set myself last week, I decided to spend my weekend thinking about append-only logs.

(If you're not interested in what these logs are, feel free to skip to the Finally, append-only notes! section)

So what's that

A log is a list. An append-only log is a list you are only allowed to add to. That may sound trivial, or constrained, but this simple property comes with a lot of benefits.

1. You never lose data. Because you're only adding to the log, you have the entire history always.
2. You can easily share a log. Because you're only ever adding data you can just send out new updates without worrying about conflicting states.
3. You can create programs that read over your entire log, and tell you different things about it

All of this sounds extremely general because a log is extremely general. You can use it to model all kinds of different things, a todo-list, a chat-room, or, an entire database!

In a traditional db, when someone does an action, say I enroll in a drawing course, the database updates the property "Jared's courses" to include drawing. If I later unenroll, that property is updated to remove drawing. If anyone want's to lookup my courses they just check the databases value for "Jared's courses".

A database built on an append-only log works different. When someone does an action, it's written to the log as an event. So the "Jared enrolled in drawing" event is logged. If I later un-enroll, a totally different event "Jared unenrolled from drawing " is logged. So we get the advantage of the full history immediately. But, to know the current state (i.e what courses am I currently enrolled in), we have to read the entire log, which sucks.

We get around with the third benefit of append-only logs. We can write a program that reads the log, and then outputs the current courses people are enrolled in. Then it listens for new events and updates it's state accordingly. We call this program a view on top of the log.

So what's any of this good for?

You wouldn't be wrong in saying we've taken one thing, the database, and made two, the log and the view. Well, the neat thing is we still keep all the benefits above! We can create databases that we can easily add views to all the time, looking at the data in new ways, while still keeping the whole history. And these views can be constructed independently, even on entirely different machines, as the log can be shared trivially.

These properties make append-only logs most excellent for peer-to-peer systems, and they're used by many. Urbit, which I talked about in #15 is one example. Hypercore is another. And radicleand ssb and orbitdb are some more. ¹

So why do I care? Well, I think the properties that make it useful for building peer-to-peer applications, also make it extremely suited for building a notetaking system!

Finally, append-only notes!

All the previous properties still stand, we've got history, sharing, etc, but I especially find appealing the ability to define views that operate on top of the note-log.

For one this is just an extremely clean mental model to have. It let's you separate out the data you want to store and the insights you want to extract from it. You can just be in the flow state while you're writing notes, and later craft your views to do different things. Back-links, a task manager, CRMs, as programs that run independently over your log.

Also, thinking about your notes as a log quite nicely fits existing mental models. Folks are already used to streaming out tweets ², and writing in journals, after all!

What am I gonna do?

Anyways, just slapping on an append-only log does not a notetaking system make. There are still a bunch of open questions:

1. How can we make it as easy as possible to create new views? If excel is the end-user version of a traditional db what's the end-user version of an append-only log?
2. How can we leverage types in both the log and the views? How can this support point 1?
3. How can we best handle events that update old values? You want to do it in some kind of standardized way so that each view doesn't have to handle it itself.

The append-only log was the missing piece of the puzzle for me in moving forward with fancynote. I had been wondering how I was going to handle the data pipeline of notes, storing them, querying them, and this seems like a great answer.

Also, I'm feeling pretty chuffed as my previous work on parsing and executing fancynote notes doesn't change at all! Wooo!