Making Sense of Sense Atlas
The goals and principles underpinning my nascent organizational cartography kit.
The previous newsletter was about the respective origin stories of Sense Atlas, my nascent organizational cartography kit, and Intertwingler, the application server that makes Sense Atlas possible—the actual heavy technical lift. So while the last newsletter was kind of a catharsis of how these both came to be, today’s is something of a meditation on the goals and principles that motivated this work.
Just The Facts
A mantra that I’ve expressed over the last few years is that stories are essential for persuasion, but if you’re already well bought in, then what you need are facts. The actual recipe, not the blog post advertising it to the Google algorithm. The times this particular movie is playing at which theatre, along with how to get there. In business we have these kinds of things as well. To dramatically oversimplify in four snappy, alliterative bullet points:
- Projects are things you do once. If two projects were perfectly identical, you could just reuse the result of the previous one. There is also an element of risk—whether the project will be completed on time and on budget—and whether the outcome, assuming you get there, will perform as anticipated.
- Products are the outcomes of projects. They form the basis of economic relationships: you to your customers, your vendors to you, and your competitors to (those who could be) your customers. Products define capabilities, and are increasingly shipping with strings attached. Mapping out the space of products tells you a lot about the directions you (and others) can move.
- Policy & Procedures are things you do over and over. They are either periodic routines, or conditional on events—“when this happens, do that.” It is useful on its face to gather these (like recipes), concentrate them (like a cookbook), and organize them based on when to use them, the inputs you’ll need to be successful, and (ideally) why.
- People are all around you. They are your customers, coworkers, vendors, regulators, competitors. People organize into (surprise) organizations, organizations make products, and products—including products like policies and rulesets—shape social and economic relationships. People live for decades, move from one organization to another, form relationships, and create all kinds of products. As such, people are the largest common factor.
Now, it just seems manifestly valuable to map these. To connect them together. In fact, it boils down a lot of what I do for a living. I’ve been accomplishing this primarily in the form of documents, spreadsheets, and slide decks. Fragmentary projections of the underlying factual structure. So much value gets left on the table, because it hasn’t been viable to deliver the structure itself. So I created Sense Atlas to fix that.
Reality-Based Structure
In software—especially business software—there is typically some kind of domain model that defines its scope:
- CRM
- accounting
- project management
- intramural messaging (email, Slack, Teams…)
- knowledge base (SharePoint, wiki…)
- myriad enterprise content management systems…
…plus an extremely long tail of purpose-specific information infrastructure (bug trackers, support ticketing, analytics, and so on), and an even longer tail of ad-hoc and appropriated systems (shared folders, spreadsheets, spreadsheets, and spreadsheets). In reality, however, things can relate to each other across thematic domains. Sense Atlas is designed to be a space where you can map connections between elements that would ordinarily reside in different silos.
In a conventional software application, the concrete way its data is represented is tightly coupled to the domain model, so it’s a big deal if you want to introduce a new kind of object, or new way that two kinds of object can be connected. In Sense Atlas, this kind of endeavour is relatively cheap. Sense Atlas achieves this by normalizing the data into an elementary structure that is uniform across all types and relationships. Once everything speaks the same language, it’s easy to represent a much wider range of connections.
Now, I’m not suggesting Sense Atlas replaces these systems—it’s not an “everything app”; that’s not what I’m aiming for. Rather, it’s a space where information can mingle.
Rapid Accumulation
Twelve years ago, I created the original prototype—the one Sense Atlas grew out of—to support my consulting work. One thing I do in that work is help teams within organizations wrangle the various concerns of their projects, as well as capture the concepts in play, so that everybody uses the same words to mean the same things. Both of these kinds of entity exhibit a (potentially complex) graph or network structure, and at the time there were no tools available to adequately represent them.
I set out to create something that accumulated information with the speed of an outliner, but without the constraint of a strict hierarchy. It was important to be able to chain together both diverging and converging lines of reasoning—and the concepts they were manipulating—with the same degree of effort it took to compose a tweet.
There has since been an explosion of tools made to do this kind of thing, usually focused on personal (and not-strictly-personal) non-hierarchical note-taking. I’ll get to why I don’t use them in a moment.
Incremental Place-Making
The word for a particular challenge of so-called “knowledge work” that is the most apposite, if a little crass, is constipation. Stakeholders (clients, bosses, etc.) tend to want to see things moving forward. So much of the progress is invisible, largely taking place inside the minds of the people on the team. Because of this, there can be periods of weeks or even months with very little, at least legible to stakeholders, to show for them. As such, evidence of progress has to be synthesized. That is, at least in my experience, you have to set aside a non-trivial chunk of available resources you could otherwise use to actually make progress, to create and present artifacts (documents, PowerPoints…) whose only function is to show that progress is being made. Not only does this (ironically) slow things down, the sheer magnitude of the diversion of attention means it can’t happen too frequently.
That, or your reports will read something like “Well, we would have made progress, but we spent all our time since the last report working on this one.”
The flipside of this situation is that actionable insights have to wait for the reporting period to be made available. If only there was some way to surface the progress in real time, like how I said before: clients of construction projects can visit the job site and watch the progress happening right in front of them. Now, I’m not so naïve as to think we can eliminate reporting entirely, but I am sanguine that the amount of the schedule it occupies can be reduced by effectively creating a virtual “job site”.
That is, you point to your instance of Sense Atlas and say “here is the place to look for signs of progress”, and it pulverizes your insights down to their individual elements and gives each one its own webpage. That makes each unit of thought a discrete, identifiable, measurable, countable artifact, that exists objectively out in the world—not just in your head, not just on your private computer. You can give its address to stakeholders—paste it into an email, text, or one of zillion other ways we have evolved to chat over the internet. Embed it into a document, or tweet it out on social media. Once people step through the structure a few times, they’ll understand how it relates to progress, and eventually come to interpret the accumulation of the structure itself as progress, without having to pore over it.
This characteristic of fine-grained addressability, that each individual element could be located, shared, and clicked on, was fundamental to the design.
View From 30,000 Feet
When I was doing the initial research for eventually became this product, I came across some discussion about how in hypertext systems—which is what Sense Atlas ultimately is—there is a tendency to get disoriented. “No Sense of an Ending” was the title of the chapter, and sums up the problem succinctly. The issue, like the text adventure games of yesteryear, is that you can’t see any farther than the adjacent rooms. This (a feature, since most of these games took place in dungeons) made it easy to get lost, and hard to tell if you’d covered the entire space. Given that, a map of the territory—along with the all-important YOU ARE HERE—was a no-brainer.
How to draw maps for structures like these is, unsurprisingly, an object of vigorous study. They take a lot of work, and you can’t really tell if they’ll be any good until you try them. I am currently on my fourth attempt, and I think that I have something that, while still rough (the geometry is definitely off in places, and I’d love to animate it), is headed in the right direction.
One thing I do like about this layout is that it’s very easy to tell when you’re off in the weeds. Click through to watch me use it to plan some recent performance improvements.
I ultimately went with a radial design that automatically sorts the structure into regions, which you can see sit largely separated from one another. The second dimension, the radius, signifies the “top-ness” of the element. That is, broader concepts and more everyday concerns end up near the centre of the map, while the more esoteric or specialist stuff gets pushed out closer to the periphery. Since there’s always more detail the farther out you go, I use hyperbolic space, so the distances get longer the farther away you get from the centre. I then smoosh this around so that the node that is the subject of the page is always the dot in the centre of the circle.
Now: one criticism I’ve gotten is that the visualization is too complex. Which is curious, because it isn’t doing anything more or less than showing you what you put in there. Notwithstanding, I can imagine implementing a “horizon” that culls the nodes that have been determined (through some yet-to-be-defined mechanism) to be “too much detail” for certain onlookers to palate. That’s coming, but I’m going to need more sample data to get there.
I am actually super excited to try alternative views of the data for different contexts. (Say a timeline, for example?) Now that I have the core online, and I’ve shaken out most of the major burrs and inefficiencies, I can turn my focus to that kind of thing.
Permeable Membrane, Reusable Data
A tool for drawing directed graphs is not innovative on its own. What is innovative about Sense Atlas is that it foregrounds priorities that other tools do not. Its core thematic module, a thing called an issue-based information system, which is used for policy analysis and design rationale, is almost 60 years old. Digital implementations of IBIS, furthermore, date back to the 1980s. Which prompts the question: If these tools are so powerful, why isn’t everybody using them?
Again, and I touched on it last week, I believe this has to do with a certain level of impermeability on the part of the implementation. I called this the then-what problem. That is, you spend all this effort getting all your relevant information into the tool. Great. Then what?
A few months ago I did a talk for the Information Architecture Conference (which you would have known if I hadn’t been buried in code instead of doing things like writing newsletters) in which I covered an analogous problem, by relating an anecdote from a project I did a few years ago. The gist was that we had done an impromptu exercise that yielded some data that would have been extremely valuable as an input to some instrumented process for project planning, if only we had had the budget to extract it. In a similar vein—and the central argument of the talk—was that people in information architecture and user experience design roles could be delivering work product that could be consumed directly by software developers, instead of requiring them to transcribe it.
Take, for example, the humble spreadsheet; the lingua franca for data exchange, and its lowest common denominator. It may surprise some that spreadsheets—and their stripped-down CSV counterparts—are a pain in the ass to assimilate. The reason why, is that there’s no schema that tells you (the programmer, or more accurately, your program) what anything is, and so every spreadsheet needs a bespoke translation table, hand-crafted just for it. Changes to the spreadsheet’s structure, of course, invalidate said translation table, and require it to be updated as well. The result is a brittle hand-off process—from non-programmer to programmer—that, all at once…
- profoundly undermines the value of the data,
- by generating additional work for the programmer,
- and, because of this additional work, creates an incentive for programmers to deprioritize processing updates.
This is also discounting all the vagaries of hand-entered data, such as numbers and dates, the potential for typos in controlled vocabularies, that kind of thing.
When you work upstream of programmers, and you produce deliverables that they first have to transcribe into something they can use, that transcription process becomes a line item, an extra chore that somebody has to do every time you update on your side. To give a contrived (is it though?) example, suppose you’re nominally in charge of the website navigation at your company. You deliver it to the dev team on a spreadsheet. This may be fine every once in a while, but let’s say the marketing team is indecisive, and there are certain points where you’re pushing updates several times a week. Is the dev team going to be on board with that?
My point here is that your deliverables should be such that you don’t have to negotiate that kind of interdepartmental choreography at all. The solution, as I argued, is to deliver work product that can be directly consumed, say, through an automated pipeline, without any alteration whatsoever. Or, at least such that the programmers can create a transformation process once that they almost never have to change. One where they don’t have to worry about errors in the data because the parts and pieces have been sanitized and normalized on input. Sense Atlas was designed from the ground up to speak this language.
I should also add that I’ve tested projecting Sense Atlas data into spreadsheets (or rather its precursor’s, but it’s the same data), and concluded that is impossible to simultaneously produce something that both faithfully reproduces the structure, as well as be usefully read, understood, and subsequently manipulated by a human being. The network structure created by Sense Atlas—or any knowledge graph editor—is just too hairy for a spreadsheet to handle.
So we have this idea of porosity, of permeability, that dramatically reduces the cost of getting structured data out of Sense Atlas—to address the then-what problem—but also helps when you need to load a bunch of data into it. More than anything else, Sense Atlas was designed around the assumption that it may be neither the first, nor the last stop for your data, which may come from somewhere else, and which may go on to drive some complete other process where Sense Atlas is no longer involved.
Data Dog-Food Double-Down
I not so much as alluded to the fact that the recent crop of note-taking products don’t do the job of Sense Atlas, as I outright said it. So I suppose I should articulate why.
The reason why typical PKM and/or note-taking products don’t suffice, is because they aren’t typed. That is, they generally don’t have node types to speak of, and they certainly don’t have link types. That is, while they do exhibit the appropriate topological structure, you’re limited to only one type of entity called note, and only one type of connection called link. And this sharply limits what you are able to say.
Node and link typing, to give one mundane example, is the reason why it’s possible to ascribe different colours to network visualization. It’s the reason why it’s possible to tint the entire page based on the entity you’re currently looking at, and it’s the reason why different pages get subtly (or not-so-subtly) different UI.
Sense Atlas—or more accurately, Intertwingler, the application server that powers it—is designed to operate over structured data with well-defined semantics. It’s because of these well-defined semantics that we can easily manipulate the Sense Atlas website to produce an picture of its contents. Another way of saying this, to those who might understand it, is that Sense Atlas (again, or rather anything running on top of Intertwingler), is its own API.
Structured data is where the value is, because it can go on to power other processes outside of Sense Atlas. You can just as easily create that data elsewhere and load it up in bulk. Indeed, I characterize Sense Atlas as “barely software”, because all it does is put a thin veneer over that very same structured data.
Anti-Platform Design
Sense Atlas is not meant to be an Everything App™, but it is intended to cross everything-boundaries. I made it to concentrate and coalesce information, but not in a way that it has to be responsible for all of it. The only way to do this is to have it converse in a language that it doesn’t fully own.
Sense Atlas—and Intertwingler, by the way—are open-source, and the data they trade in, which you can extract one hundred percent of, is all open-standard. This is an insurance policy that you will be able to glean value from the content you put into Sense Atlas in perpetuity, no matter what your relationship is to me.
If I had to pick one single most important aspect of the design of this product, this would be it.
So There You Have It
As I noted last newsletter, Sense Atlas is serviceable for supervised operation, and I’m actively using it to plan its own development—which you can see happening in front of you at senseatlas.net. I’m offering it for the first year of operation to consulting clients as a free add-on to my portfolio of services. If you run a team, and you need help disentangling a project, gaining visibility into some infrastructure, or implementing some policy, do get in touch. If you’re on a team facing one these challenges, consider forwarding this newsletter to your manager.