September 9, 2023

Pivot

Now that the Summer of Protocols is in the rear-view mirror, it's time to tee up the rest of the year.

The Summer of Protocols program has concluded and my deliverables are in the can. Because the program’s ur-artifact is going to be printed, and a GitHub repository is not especially conducive to that, we decided at the last minute to redefine my “protocol artifact” to be the README file that adorns the repository’s cover page—​especially since the software it’s attached to is almost certainly going to be something I tinker with for years to come. As such, I diverted the last couple days of the program to rewriting that.

Subscribe now

Intertwingler itself—​the cheeky Nelsonian moniker I ultimately ended up calling the piece of software I’m writing—​is coming along. To refresh, it’s an engine for making websites, in the sense that WordPress is an engine for making websites, but beyond that it’s unlike WordPress or anything else in the category. Last weekend I got the transforms (more on these in a second) working as stand-alone services, which further vindicates my decision around mid-July to go with the more ambitious design, even at the ultimate expense of being “done on time”.

I put that in scarequotes because while the original plan expressed a relatively clear understanding of what “done” looked like, the mid-July pivot muddied that definition considerably. Just as well, though, as my initial, more conservative plan wouldn’t have been as useful. As I mentioned elsewhere, there are five distinct constituencies who have interests in this thing, and balancing all the equities equates to a lot of moving parts—​on top of an already large number of moving parts of the technical implementation itself.

To save you a click, the five distinct constituencies, organized by increasingly esoteric niche, go like this:

1. End users interact with an Intertwingler-powered website just (well, hopefully not just) like any other website.

2. Penultimate users will instantiate existing applications made on top of Intertwingler—​once some exist—​on their own computers or hosting services. These in many cases overlap with end users.

3. Proximate users will use Intertwingler to make said websites and applications. These make up the initial target group.

4. Extension developers use the framework and protocols I designed to make Intertwingler more powerful and do more things.

5. Porters use Intertwingler as a reference implementation and apply the principles it encodes to translating it into other programming languages.

So while “done” is a categorical misnomer, we can say that the “milestone” of having something that is serviceable to the aforementioned proximate user looks like a command-line tool you can install off a package repository (I will also be generating a Docker image) that will spawn an instance of Intertwingler that you can start sculpting into a website or app. The resulting artifact is still going to be early, early alpha quality, and preparing all the tutorial documentation and onboarding materials is going to be a whole other project unto itself.

Anybody using Intertwingler at this stage should also expect things to break. In particular, I will be going after the storage mechanisms (two other, separate pieces of software) to make them behave more sensibly, and I already expect the changes I make to break any instantiated databases, so don’t put anything in there you don’t have a backup of. There are also strategic indeterminacies and other tentative decisions that I can’t tell whether they paint the system into a corner or not until I get the whole thing up and running and poke at it a bunch. Early adopters, consider yourselves warned.

The main substantive difference spurred by the mid-July pivot, aside from the cheeky new name, is primarily of interest to the last two groups: the extension developers and the prospective future porters of Intertwingler to other languages. Namely, it is the consolidation of all functionality into a single conceptual entity I am calling a handler, which I am appropriating from the lingo of Apache modules—​although my handlers work slightly differently. Everything in Intertwingler is a handler, which is a microservice that responds to one or more URLs via one or more HTTP request methods. A handler is a directly-callable object that expects some facsimile of an HTTP request, and returns a response in kind. For the extension developer, this is an extremely well-defined target. It also means that any handler can be run as a stand-alone service, which is not only convenient for bench-testing, but has strategic importance that I will touch on momentarily.

The other major idea, which was always part of the plan but originally not as audaciously, is the notion of transforms. These are special handlers that consume an HTTP message body and return a modified body which is grafted onto the message (request or response) and sent downstream. Transforms can therefore be daisy-chained to one another to create complex filter pipelines to manipulate pages and other media.

Because “everything in Intertwingler is a handler”, and every handler is a potentially stand-alone microservice, this means that transforms are also microservices which can be configured to stand alone. Since they speak HTTP, they can be written in any language, and do their communicating over the network. This comes in handy, for instance, when you want to delegate certain loads to dedicated infrastructure (say, special hardware like an ASIC or GPU). The other goal with this design is something I’m calling intelligent heterogeneity—​preemptively addressing the fact that no system remains confined forever, exclusively to one technical stack. The idea is to make the system porous enough so that different programming languages, frameworks, and hosted services can interact with one another by being coupled in the loosest possible way. The only requirement is that the handlers conform to the protocol.

What I’ve been working on this week, then, is the configuration language for all these handlers and transforms and stuff. I anticipate the main response queue itself will have about a dozen transforms in it alone, to do various things to HTML as it passes through. Statically-configured transforms will also have to be handed their parameters somehow—​something I anticipated when I wrote a Transformation Functions Ontology three years ago. That takes care of part of it, and I started roughing in an Intertwingler Configuration Vocabulary earlier this week to take care of the rest. Part of the motivation for doing it this way is so that you can configure Intertwingler, using Intertwingler—​that is, online through the Web interface. There probably will still be a configuration file, but it won’t say much except where to look for the rest of it.

This configuration language will enable me to create the spine of Intertwingler, which is the next step. After that, it’ll be time to package it for initial release. I will be releasing the essay—​the other half of my Summer of Protocols deliverables—​but not until after the organizers do. Follow the Protocolized newsletter for when that happens.

As for moi, I have this and a couple other commitments, but definitely have some time in Q4 to allocate. If you or your employer are doing any protocol stuff, or any really gnarly Web stuff, or your project just needs another brain on deck, give me a holler.

Subscribe now