Well, it has been a while. This newsletter marks the open sourcing of my distributed S3 clone github.com/justincormack/argmin, that I have been working on for, er, six months now! This is quite a long time for an experiment, but times are interesting.

The aim of the experiment was driven by the wave of experimentation with AI code generation. Building small things was fun, but I was more interested in whether we could build things that were more ambitious, like a large scale distributed system. These normally involve teams of people working for many years, and would be the kind of thing that generally you would avoid building because of that. But AI supported software engineering is much more interesting if we can use it to build things that were very hard to build before, and to build much larger systems, rather than just building small things faster.

While there is a lot of code (although in line with the other implementations; this is largely a necessary complexity problem), the output is currently a single 34MB binary with no outside dependencies, and the source code is reasonably easy to work with, although it always needs more refactoring and cleanup. The principal requirements internally are SQLite, OpenRaft and Rustls and the HTTP stack.

I talked about building this, especially the testing side in a talk and podcast

• AI Native Dev Con: When Tests Lie: Using Observability to Keep AI Honest
• Asynchronous and Unreliable Podcast with Anne Currie

Overall, the engineering role becomes not reading the code (950k lines of Rust so far, much of it tests) but finding ways in which it might be wrong, and providing direction and tools. Oh and hardware, like the cluster of mini PCs on a programmable PDU for testing live power off failure recovery.

Right now, as the README makes clear, this is not quite ready for production use. It is missing the kind of production tooling and observability you need for production, and I have not enabled support for schema upgrades, although all the framework is in place now. However it is close, and the next stage is to work on this transition, while also continuing more detailed distributed correctness testing. As this is a storage system I am pretty paranoid about the level of correctness needed to not lose data, so the emphasis has been on safety not performance, so there are no options to not fsync, and there is a paranoid amount of checksumming, and making invalid states unrepresentable.

In terms of fidelity to the AWS interface, I think this code definitely does a better job than other implementations. The most comprehensive test suite before this was Ceph's S3 tests which most other projects use. It has about 800 tests, and is very slow to run. The suite I have is almost 2000 tests, and is so high fidelity that sometimes I have to tell the agents that they are running locally not against AWS. It even matches the spelling errors in error messages. It is easier to have a good oracle than to use judgement in when to diverge, although there is a policy of having correct error messages not just generic 500 errors to make it easier to detect real bugs. There were also some security findings that matched upstream S3 behaviour and are replicated, even though they are certainly mistake prone.

This was always intended to be open source, once I decided that I could make it good enough, there were plenty of people saying that AI could not code in large codebases and things would all fall apart. I was not sure that the transition from a single node project to a distributed one, which took months and 13k lines of notes by the AI to keep track, would work. I still feel that intense supervision is needed, and different kinds of work. I recently came across this 2009 piece on automation with the diagram of what people want and expect versus the real outcome

in Bay's words, "force multiplication through accomplishment of more demanding tasks." Humans are always taking on challenges that are at the edge of the current capability of humans and machines combined.

In this era of lack of trust and slop, open source is more important than ever. And if the code is not valuable in itself, why not make it open? As I talked about some time back (talk not online), proprietary code is going to discover the things open source has always known, that it is the things around the code, such as community, that are more important than the code itself.

I still think we can be in a golden area of new software, especially systems software, and the opportunity to build secure, correct software at scale is available to more people. It is still not easy, and there are no shortcuts, but it is time to build!