What Do You Do When a Piece of Your Stack Goes Bad?
The risks to your software supply chain, especially these days, include a lot more than just malware.
One of the things I have been busy packaging into a service recently is what I’m calling a Supply Chain Stress Test. This service started out with a company I worked with a few years ago who was angling to internationalize their whole organization. Doing so entailed looking at the various software-as-a-service vendors that supplied capabilities to their website and product line, and document the extent to which their products supported messaging in multiple languages. A lot of my work boils down to making indexes and inventories of things, so this one included:
- the vendor and product
- contact information for the vendor’s rep(s) for my client, if applicable
- the service tier, if applicable
- the actual assessment of the capability my client was interested in (in this case, localization)
- recommendations for how to proceed
- links to the relevant product documentation, viable competitors, etc.
…lather, rinse, repeat.
This kind of analysis, however, is by no means the only reason to do this kind of work. Assessing your SaaS vendors’ capabilities is one aim, to be sure (ideally before you contract with them, but these guys didn’t start out with this requirement), but these relationships are also not without their risks:
- They go down, disrupting your business. Even stalwarts like Salesforce and AWS (and anything running on top of it) seem to go down for a period of a few hours around once a year. That may be too much for you!
- They can get hacked. Most, if not all, of a SaaS vendor’s security problems, are also your security problems.
- They can just plain nuke your data. This doesn’t happen too often anymore, because the industry generally knows better, but it can.
- They can go out of business. This is like going down and nuking your data, permanently. (At least until your data shows up again in the bankruptcy auction.)
- They can change their terms. Software companies deal on a take-it-or-leave-it basis, and tend to change the deal on short notice. Maybe the terms were fine when you entered the relationship, but what if they become unacceptable?
- They can get bought out. And absorbed, or terms changed, or prices hiked, or the acquiring company is your nemesis, et cetera.
Now, it seems that there’s another risk to consider, and that is your association with the vendor becomes a liability. The basis could be as calculated as a matter of brand safety, or it could be that you genuinely just don’t trust them anymore. What’s it going to take, then, to get out of that relationship?
Strictly speaking, this situation has always been on the table, it’s just that it seems a heck of a lot more front-of-mind now.
The other contributing factor to writing this newsletter—besides hawking my wares—is a couple current events, one that affects me somewhat, and another that thankfully does not.
The first of these is that a sequence of ultimately political decisions has resulted in the hijacking of RubyGems, the package management infrastructure for the programming language Ruby, which, for boring pragmatic reasons, happens to be what Intertwingler is written in. The gist of the drama is that the stewarding organization for all of this business invited a positively radioactive individual to keynote their flagship conference, provoking both resignations and a revocation of funding in response, and the ultimate hijacking of the intellectual property by the org’s remaining members in response to that.
Said individual is none other than David Heinemeier Hansson, also known as DHH, who has always had a reputation for his attitude, but whose mouth noises in recent years have begun sounding increasingly—and unambiguously—white-supremacist. DHH invented the Web framework Ruby on Rails, which is what put Ruby on the map. (I don’t use Rails for Intertwingler; I actually kind of compete with it.)
Thankfully I haven’t had to do anything in response to this—I already have my hands full dealing with Google unceremoniously vaporizing XSLT—and probably won’t have to do much, but it was enough to get spooked there for a second.
The other thing that happened this week was that the CEO of the cloud services company Vercel, Guillermo Rauch, circulated a selfie he took the other day with Benjamin Netanyahu. The photo was captioned with some pablum about AI for peace, because why not. An especially charitable reader might rule this a PR own-goal of legendary magnitude, but it looks to me like he knew what he was doing. Naturally people are pissed and they’re fleeing his company’s services.
Vercel proper is ostensibly not too difficult to quit, but what is difficult is another thing with Rauch’s name on it called Next.JS. This is another framework in the vein of Rails (and to some degree Intertwingler), and ripping it out (much like Rails…or Intertwingler) would basically mean throwing out most of your codebase and starting over. Some people are threatening to do this; I doubt many will. What I find interesting is the fact that this kind of thing is getting more consideration than I ever remember.
I realize I’m now talking about open-source infrastructure now instead of SaaS vendors, but I think the effect of an unseemly association is qualitatively different somehow. For starters, I think it’s a lot more “intimate”, for lack of a better term. Like, to the extent that we have any contact with the authors of open-source projects at all, we platform them on podcasts, we give them keynote slots, we go to long-table dinners with them at conferences. We hang out with them in the bar. Sure, he may be an asshole with a monumental ego, the attitude goes, but he’s also the guy who made the thing we all use for our livelihoods.
For a long time, the norm seemed to be that (putative) geniuses should be accommodated in their …idiosyncrasies. It seems like this may be changing? For one, the truly awful people seem to be leaning into it, which makes them easier to spot. For another, the #MeToo era appears to have spurred the development of a modicum of antibodies. People are now proactively weighing the governance of these assets so that one degenerate individual can’t sour the entire endeavour. It’s now another dimension along which these projects compete.
I am thinking in particular about the people in the Rust community, who have demonstrably given a lot of thought to how an open-source project (in this case, the Rust programming language) ought to be governed. You can see this conscientiousness everywhere in their culture and their product.
And that’s just it: there’s plenty of competition in open-source infrastructure now—at least there is a lot of the time. You don’t have to pick the one helmed by a person who is potentially dangerous.
Why should you care? I mean, aside from the association being bad for the brand? Well, there’s the practical matter of your own influence over the project, that it moves in the direction that aligns with your interests, which manifests at the lowest level as getting your bugs fixed and patches merged. Anything more typically involves developing a relationship with those people, and the question there is do you really want that? At root, though, using somebody’s open-source software in your stack gives them oxygen, and there’s a direct line between that and attention, influence, money, and even actual power.
Next.JS has been fantastic marketing for Vercel, just as Rails has been fantastic marketing for Basecamp.
I want to be clear, furthermore, that this isn’t a matter of ideological purity-testing—although I suppose it might be for some people—but rather a frank issue of security. And I don’t mean just that of your supply chain either, but for society at large. I’m not even suggesting something that amounts to a boycott, just taking the culture around the artifact into account, and in particular its progenitors, and politely declining to use it if it doesn’t pass the sniff test. This is about establishing a norm—which these days seem to be in short supply—that conduct is on the table, and that no product or piece of infrastructure, or leader thereof, is truly irreplaceable.
Granted, it’s easier to do this before you commit to these relationships. But if you’re looking to assess your vendors and/or stack for any reason, and if you need help sunsetting any of them, you know who to call.