Soil is a file-based, transactional object database for Pharo — it stores object graphs directly, with no mapping layer between your objects and the disk. This series builds it from the ground up: it starts with turning a single object into bytes and works outward through object graphs, identity, transactions, MVCC, indexes and crash recovery. It assumes you know databases in general but nothing about Soil's internals, and it trades grand theory for how one concrete system actually works.
Soil: turning an object into bytes
Soil: turning an object into bytes A few years ago I wrote two posts about building an object database from scratch, BYOB - build your own (data)base and BYOB - object serialization, and then went quiet and just built the thing. Soil is real now: a file-based, transactional object database for Pharo, running in production. I want to go through its architecture properly this time, from the beginnin
22 July 2026
Soil: serializing a graph
Soil: serializing a graph In the last post we turned a single object into a self-describing stream of bytes and read it back. That was the easy half. Real objects don't live alone. They point at other objects, those point at more, and before long you have a graph. Sometimes that graph shares a node from two places, and sometimes it points back at itself in a cycle. This post is about writing such
23 July 2026
Soil: identity and the heap
Soil: identity and the heap The last post ended with two loose ends. A reference from one cluster to another is stored as an "id", but I never said what an id is. And I kept talking about clusters "on disk" without saying where on disk, or how you get from an id to the actual bytes. Both ends meet in the same place, so let's tie them off together. Why a reference can't just be a position The tempt
27 July 2026
Soil: transactions and how a commit happens
Soil: transactions and how a commit happens The last post gave every object a stable id and a way to find its current bytes through the index. That's enough to read and write single objects. What's still missing is the thing that makes a database out of that: a way to change several objects at once and have it count as one atomic step. That's a transaction, and this post is about what one actually
30 July 2026
Soil: MVCC and snapshot isolation
Soil: MVCC and snapshot isolation The last post walked through what a commit actually does: serialize outside any lock, then take one global lock just long enough to bump the database version, check for conflicts, write the journal, and apply it. Only one writer gets to be in that critical section at a time - the same trade-off SQLite makes. Readers, though, were said to bypass that lock entirely
11 August 2026
Soil: catching write conflicts, optimistically
Soil: catching write conflicts, optimistically Every transaction in Soil gets to act as if it has the whole database to itself - read whatever it wants, take its time deciding what to change, and work all of that out without ever announcing itself to anyone else running at the same time. That's a comfortable illusion right up until two transactions both decide to change the same thing. Something h
31 August 2026
Soil: knowing what changed
Soil: knowing what changed The transactions post left this on the table with a one-line mention: "'Dirty' itself comes from one of two tracking strategies - either you mark an object dirty yourself, or, as of the current version, Soil detects changes on its own by hashing an object's content and noticing when that hash no longer matches." That's the whole subject of this post: what those two strat
7 September 2026
Soil: the journal, and what makes it a WAL
Soil: the journal, and what makes it a WAL The transactions post left this one dangling: "Every new or changed object, every index update, and the new database version becomes an entry in a journal - an in-memory list first, not yet on disk [...] The journal is central enough to deserve a full post of its own later." This is that post. From transaction to journal entries The thing a transaction as
11 September 2026
Soil: checkpoints and recovering from a crash
Soil: checkpoints and recovering from a crash The transactions post left this dangling: "if any of this actually changed the database, Soil checkpoints afterward - flushing everything to disk in a way that can survive a crash. What a checkpoint really guarantees is its own post." The journal post added its own loose end on top: every fragment file has to start with a checkpoint entry, for reasons
14 September 2026