April 29, 2021

65: Collaboration in Hyperspace

I want to talk about some things that have been on my mind as I’ve been implementing collaborative editing in Hyperspace this past week.

It’s been a lovely combination of strategy, technical wrangling and character development.

First, a broad overview of what I wanted to accomplish.

Hyperspace is a social space for learning. People can create and interact with expressive documents that are live and collaborative, and which can be richly networked to create both social and informational structures.

The “live” bit is what I was focused on this week. When two people have the same document open they should be able to see each others interactions immediately, and be able to interact simultaneously with the same things.

Hyperspace is largely a text focused environment, so the key here is collaborative text editing, like Google Docs. That’s what I needed to implement.

The development process

A quick overview of the tools I’m working with:

• Fauna: This is where we store the data people will be interacting with. While it’s extremely flexible, it has no built-in notion of collaboration (much like cats).
• Cloudflare Workers: Specifcally Durable Objects. This a new-ish platform from Cloudflare that is very useful for coordinating between multiple peers.
• Yjs: This is a really powerful Javascript library that implements a CRDT, (a conflict-free replicated data type). This is what actually handles all the complexity of allowing people to interact with the same data at the same time.

These tools are definitely not the only way to implement collaborative editing. Cloudflare Workers and using a CRDT specifically are both particular choices.

CRDTs are newer than other technologies for achieving collaborative editing, and are quite complicated, but they’re improving extremely rapidly, and there’s a very large amount of developer energy behind them. That’s what I’m betting on.

Cloudflare Workers is a limited platform and isn’t as widely adopted as other serverless systems. But, it’s also improving extremely rapidly, and it’s shipping unique features, like Durable Objects, that just don’t exist anywhere else, and are uniquely useful for the kinds of distributed, end-user applications I’m interested in building.

Going uphill

I started by making a little demo with both Cloudflare Workers and Yjs, which was incredibly satisfying. Quickly getting up to speed with a piece of technology is the kind of work that I’ve legitimately gotten good at over the last couple years, and seeing it all come together quickly in a small collaborative text editor felt great.

Once I had proved out the basic interactions, and knew my way around the tools, it was time to actually implement the feature in the application.

This is where things get messy. All the while I was working on this feature, I was feeling like I was doing it the wrong way. I’d make an increment of progress, and then survey my code and begin to think about the next step, and would immediately feel that I’d just been piling hack on top of hack thus far, and I just had to clean things up if I wanted to keep making progress.

Inevitably I’d be stuck for a few hours, and then arrive at another “hack” that would inch me another increment forward. And repeat.

Part of the frustration was that it felt I was forcing each tool to work with the others. This wouldn’t normally be a problem, after all integrating is largely the MO of programming. But this time I had a constant vague sense of a overarching system just beyond my comprehehnsion, or abilities.

Every time something was a little more complicated than expected, I had the feeling that if I stopped working on implementing and spent just a little more time thinking I’d be able to figure out the perfect system which would make the implementation trivial.

It’s hard to describe how split my mind is about this. Even as I’m writing this, and justifying my decision to you a little part of me is going: “well if you just stop and think, you could figure it out right now, and you could write about the solution in this newsletter”.

I’ve learned to discount that voice heavily. Spending too much time abstracting early¹ has killed so many of my projects, so with this one I just kept doubling down and focused on implementing features.

And, not-surprisingly-surprisingly, it paid off. It turned out that there were always hacks that were a little cleaner than the way I was worried I’d have to do it, and the show could go on.

Collaborating in Hyperspace

There are a couple of things that potentially make collaborative editing trickier than usual in Hyperspace. The first is that the vision for the application itself is as a flexible, expressive environment. In order to achieve this we have a very abstract, flexible data layer. The collaborative systems we build can’t just let people collaborate on a few, isolated things, but everything that can be expressed in the data layer should be collaborative.

Doing this properly, requires abstracting out the collaborative layer into a more generalized system. We don’t need it right now, but we most likely will one day.

Ultimately, this is my core tension: I beleive in protocols over products. In general, but especially for learning. I think people should be given tools that can use and manipulate on their their terms, and for their needs. But, we need to make products that are useful to people now.

Keeping both of these goals in my head get’s pretty messy.

We need to be with each other

So, to finish up, why does collaboration matter so much?

Ultimately we want Hyperspace to be a tool people can inhabit together. When you do something in a space, other people should be able to see it, and respond. The actions you take need to be not only mechanisms to achieve your goals, but a medium to communicate with others. This means we have to support both persistance and real-time.

It’s not enough to open up a zoom call to talk something out, and then split up and do your work seperately. Our tools need to be able to handle the whole temporal spectrum. You should be able to get in the zone deep in your own thoughts, and jam with others, in the same space just as you would in an office, or a classroom.

---

P.S Thanks for reading! I’m feeling pretty energized after the lil break, though I think I’ll still be refraining from the big newsletter batched topics. For now I’ll stick to these looser ones, most likely about the things I’m actually working on, week to week. See you next time!