Twenty years of S3

March 14 2006, twenty years ago today, marked the launch of Amazon S3

Amazon S3 is storage for the Internet. It's designed to make web-scale computing easier for developers. Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

I talked before about how S3 was foundational in my experience of the cloud, and how it should work. Simple. Just "malloc for the internet" as Jeff Bezos originally asked for.

The original S3 was much simpler than the current one, with no multipart objects, eventual consistency, and many fewer features. Over time, the complexity has grown, and there has been a rewrite, directory buckets, that don't really replace general purpose buckets. And S3 has become a standard outside AWS, with other cloud providers mostly copying it, and many other implementations.

So as my 20 year tribute, three weeks ago I started seeing what happened if I use AI to implement an S3 compatible object store. Its been an interesting journey so far, I now have 86k lines of Rust code, two thirds of which is tests, and very close matching of AWS behaviour for the large amount of the API surface that is implemented. It is missing full multiuser ACL support, encryption support, and so far it is single host although some of the structure for multiple hosts is there. I wanted to see what AI tools could build at scale, and how to get them to build the architecture and quality I wanted.

Having an external source of truth, letting the AIs use my AWS account for testing is very useful, keeps it grounded, and costs almost nothing in API costs. But that only gives you observability of things visible in the API, it is hard to find faults, like for example issues such as TOCTOU which turned up the other day, although some of the race condition tests did trigger on integration tests. So far I have found most of those issues by inspecting interface contracts and code flow walkthroughs, which do not scale very well but does keep me in the loop to understand what is going wrong. You really need tests over internal behaviour, tests over an observability layer, in order to make more progress, so that is what I am working on next. You need this observability to give you parallel models to the public API which show the internal models of what is happening.

Once you stop reading all the code in a project, because there is too much of it, there are two ways to work. The first, which I have mostly been doing for the first three weeks, is to build a mental model along with the AI build, watching every move, understanding the scope of changes and intervening and directing. Compared to when you write the code yourself, there is a slightly lower fidelity as you miss some of what happens, but the speed of work and iteration actually helps in some ways too with the sense of immersion and building your own mental models. I have been reading David Bessis excellent book Mathematica about how to do mathematics (thanks to Cedric Chin for bringing him for a great members only discussion with the CommonCog community), and find a lot of overlap in how to think about this work, your aim is to develop your own intuitive mental model of the deep structure of the architecture while the AI is doing the "system two" mechanical work. I will write about this more later, but I recommend David's book or his Thinking fast, slow, and super slow as a starting point.

Making deeper models of programs as black boxes, and tools to work with them falls out of this naturally. Once our mental models are more complex than surface behaviour, we need to materialise these models, both for our own observability, and also so we can use these to give the AI the right feedback loops to design around. This naturally leads to lightweight formal methods and full formal methods, and also the use of testing technologies such as deterministic simulation testing, as with Antithesis. I remember saying a few years ago that AI driven software development could lead to a rise in quality as we have the ability to use more of the tools that we have and can build, and I do think this is right for what Charity Majors has called durable software.

I will let you know how this goes in the next few weeks soon. And yes I will open source this when it is in a better state.

Anyway, happy birthday S3, enjoy your bucket of presents. It is an interesting time, with war testing data durability and storage prices surging.