Nature

There are a number of hornbeam woods near where I live that I like to walk in. These seem to be quite old, and perhaps part of larger historic woodlands.

New Rusty object stores

I recently started looking at two relatively new object store implementations, both written in Rust, but very different. As you probably know, I love object stores, check out my talk from Kubecon a while back if you want to understand why

Rustfs

Rustfs has a website at rustfs.com and source at GitHub rustfs. It was launched as a public project in the middle of 2025, although it was started a year earlier. It is in a beta process for a 1.0 release which is estimated to be sometime around April 2026 or so. The documentation is terrible and scheduled for improvement, and should only be read to get a gist of the system and aims right now, as at least some of the details are totally hallucinated by an AI in Chinese and then translated. The code is Apache 2 licensed, although there is a CLA requiring copyright assignment, which is not ideal. The original development seems to have started in China, but there is a broader team now, but what sort of commercial organisation there is, if any is very unclear.

As a broad summary, Rustfs is essentially a Rust rewrite of Minio. It is not a line by line rewrite, but the vast majority of architectural decisions are identical, such as the way erasure coding groups are set up, and the metadata formats on disk even look similar. The top line use cases, as a store for AI in particular, are pretty similar. There is a strong focus on performance, and the strong expectation is that the store runs as first line storage on NVMe SSD, not hard drives. The expected hardware is clusters of 16 machines with 8-12 NVMe drives each, connected with redundant 100Gb or 200Gb ethernet connections, and with redundant power.

With Minio now no longer open source there is of course a lot of interest in open source replacements in this space, and Rustfs is pretty much the same operationally as Minio, so

Rustfs has a few features that Minio does not have, in particular it has an integrated sftp/ftps server. It has pretty comprehensive S3 compatibility testing, and a (node local) cache layer for performance.

There are a lot of design tradeoffs in building an object store, and I would say here performance is definitely top of mind "2.3x faster than MinIO for 4KB object payloads. RustFS is an open-source, S3-compatible high-performance object storage system" is the tagline, rather than correctness or durability. In terms of durability, it adopts the same model as Minio were if there is a failed disk in an erasure coding set, it will write the data already degraded, and rely on the recovery process to rebuild. Now SSDs are more reliable than hard drives, but you should probably have a short SLA for replacing drives at least. In terms of correctness, I opened an issue that conditional PUT is not concurrency safe after some testing. I am fairly sure that Minio does do this correctly, will be interested to see what the response is. Minio has a distributed locking mechanism which it looks like Rustfs removed a while back, probably in the interests of performance.

Conditional PUT is the only atomicity primitive in object stores, allowing building concurrency handling mechanisms, and transactions. For 18 years or so S3 did not have conditional PUTs at all, and this was fine for use cases like websites, but data stores like Delta Lake and Iceberg now rely on it; before it existed on S3 they generally used Dynamo to provide the transactional support, which is of course much messier. Now all the hyperscalar object stores provide this primitive it is becoming a requirement for newer applications. In my view, providing a broken implementation under concurrency is worse than just not providing it at all which makes it clear that you cannot use it.

I don't have a full scale operational cluster at present, so I won't comment about operational issues in general, other than to say of course it is not 1.0 yet. However it is operationally relatively simple to understand, much more so than say Ceph. Observability probably needs more work, but building up a mental model of how it works and what is happening should be relatively straightforward.

Garage

In comparison, Garage is very different, more, well maybe garagiste, as it is French. The website and source repo outline the philosophy, of being simple to deploy and operate. It is designed to run on any old machines, with SSD for metadata and data on cheaper hard drives. There is no erasure coding support, just triple replication, and it intentionally does not support conditional PUT as it is eventually consistent, so cannot make any such guarantees.

The code is developed by a couple of primary developers and a larger number of small contributors, since 2020 or so, under an AGPL license, with some EU funding.

Metadata is stored locally on each node in LMDB (default) or sqlite or a similar store, and replicated through CRDTs. You probably want to stick to the recommendation of using SSDs for metadata (eg the boot drive) and HDD for data, as 3x replication is going to be pretty expensive on SSD, versus erasure coding which is typically in the range of 1.3x to 1.67x depending on the scheme choice. With SSDs being expensive versus hard drives, any scheme that reduces storage is better. Rebuild times with replication are less than with erasure coding (no read amplification) on a small number of hosts, but can be less on larger clusters as the work can be distributed. A 10TB hard drive full rebuild will take around a day with 1Gb bandwidth (which is similar to the write bandwidth of a hard drive anyway), so you want to take that into account when thinking about reliability with this sort of system.

Data encryption support is minimal, with support for SSE-C encryption, where the client provides encryption or decryption AES keys on every operation. This has recently been deprecated by AWS and is gradually being removed, so support from tooling may not be good. For some applications, such as public websites, this does not matter, and for some others applications might want to manage encryption before sending the data to storage. Key management is a difficult issue for self hosted object stores, as there is not a single solution that is easy and convenient like the cloud providers integrated key management APIs. You can of course set up encryption at rest on the raw drives using LUKS, but that addresses different threats.

Overall it is great to see new object storage implementations appearing, and good to see ones with such a different set of tradeoffs, as that gives lots of options for different use cases. Both have made things operationally simple, which is a great design decision.