Soil: MVCC and snapshot isolation

The last post walked through what a commit actually does: serialize outside any lock, then take one global lock just long enough to bump the database version, check for conflicts, write the journal, and apply it. Only one writer gets to be in that critical section at a time - the same trade-off SQLite makes. Readers, though, were said to bypass that lock entirely and never wait on a commit in progress, which sounds almost too convenient. The obvious way to make concurrent reads safe would be to have them take some kind of lock too - a shared read lock, blocking until no writer is active, or something equivalent - but that just reintroduces the contention the single-writer design was trying to avoid in the first place, only now on the read side.

Soil doesn't do that. A reader never waits, never blocks a writer, and still sees a database that never looks half-written or torn between two commits. That combination - no locking, no waiting, and still consistent - isn't a lucky side effect, it's the specific thing multi-version concurrency control (MVCC) is built to provide. This post is about how Soil gets there: what a transaction's view of "the database" actually is, and how that view stays correct without ever taking a lock to read.

One number, fixed at the start

The database version behind this is one number for the whole database, not per object, that only ever goes up - every commit takes whatever it currently is and stakes its claim on the next integer. That's the entire timeline of the database in a single counter: version 4 is a strictly later point than version 3, and nothing else can also be version 4.

A `readVersion` is just a position on that same timeline, handed to a transaction once, at the moment it's created:

tx := soil newTransaction.
"tx readVersion is now whatever the database version was
 at this exact moment, and it will not change again"

Because the database version is that one strictly increasing counter, fixing a `readVersion` really does mean fixing a point in time, not just a number - "the database as it was at version 4" is unambiguous in exactly the way "the database as it was at 14:03:07" isn't. That single number is the transaction's whole notion of "now." Every object it reads, for as long as it lives, gets resolved against that fixed version - no matter how many other transactions commit in the meantime.

A chain of versions, not one set of bytes

This is multi-version concurrency control, MVCC - the same idea Postgres runs on: never overwrite, keep several versions of a thing around, and let each transaction pick the one that matches its own snapshot. Soil applies that per object rather than per row. The heap that stores object bytes is append-only - a commit never rewrites a byte that's already on disk, it only ever adds new ones at the end. A new version of an object is just more bytes appended there, linked back to the previous version's position, so each object ends up as its own chain of versions on disk rather than a single current copy - newest at the head, oldest reachable by walking backward. Nothing marks an old version dead in place; a version is just no longer the head of the chain. Whether and when older versions ever get reclaimed is its own question - the online garbage collector that Soil v5 introduced as a preview is where that kind of cleanup would live, not part of what reading needs to work correctly today.

A chain of versions on disk with a fixed readVersion walking backward until it finds one old enough

Reading "the object" therefore means picking the right link in that chain: the newest version whose own version number is still old enough to belong to the reader's snapshot.

at: index version: readVersion
    record := self at: index.               "start at the newest version"
    [ record version <= readVersion ]
        whileFalse: [
            record hasPreviousVersion ifFalse: [ ^ nil ].
            record := objectFile atPosition: record previousVersionPosition ].
    ^ record

Walk backward through the chain as long as the current version is too new. Stop as soon as one is old enough - that's the version the transaction is allowed to see. If the chain runs out first, the object simply didn't exist yet at that snapshot.

Proof in one test

The effect of all this is simplest to see directly:

object := SoilTestClusterRoot new nested: 'first'.

tx1 := soil newTransaction.
tx1 root: object.
tx1 commit.

tx2 := soil newTransaction.
tx3 := soil newTransaction.

tx2 root nested: 'second'.
tx2 markDirty: tx2 root.
tx2 commit.

self assert: tx3 root nested equals: 'first'

`tx2` and `tx3` both open after the first commit, so both start out seeing `'first'`. `tx2` then changes it to `'second'` and commits. `tx3` was already running before that second commit happened - its `readVersion` was fixed before the database version moved on - so it keeps seeing `'first'`, even though a newer version now exists and even though `tx3` never touched a lock to get that guarantee.

Three transactions on one timeline: tx1 commits v1, tx2 and tx3 both start reading v1, tx2 moves on to commit v2, tx3 stays pinned to v1 and still reads 'first'

Time travel isn't a special case, it's the same mechanism

Because old versions genuinely still exist on disk rather than just being logically hidden, "give me a snapshot from a moment ago" and "give me a snapshot from an hour ago" are the same operation - only the number differs. Soil exposes that directly: a transaction doesn't have to start at the current version at all.

tx root at: 1 put: 2.
tx markDirty: tx root.
tx commit.
"root is now 2"

past := soil newTransactionForVersion: soil control databaseVersion - 1.
past root.   "-> 1, the database as it was one version ago"

`newTransactionForVersion:` pins a transaction's `readVersion` to any past database version, and every read on it walks the same version chain the same way - there's no separate history API to learn. `allVersionsOf:` goes further and returns an object's entire chain at once, independent of which snapshot the asking transaction happens to be pinned to.

There's one deliberate boundary here: a transaction pinned to the past can read, but not write. Committing from one raises `SoilObjectHasConcurrentChange` the moment a newer version already exists - which is really just the ordinary write conflict check from the last post, applied to the most extreme case of "someone already wrote after you read." The database doesn't support forking its own past.

A transaction pinned to v1 reads 1 successfully, but writing from it and committing raises SoilObjectHasConcurrentChange because v2 already exists ahead of it

The other half: writers, not readers, hit conflicts

Reads never conflict with each other under this scheme - a version chain only ever grows, it's never edited in place, so an old reader and a new one can both be satisfied from the same chain at once. The last post already showed where conflicts actually get caught: at commit time, by comparing the position a writer read an object at against that object's current position on disk. If they no longer match, someone else committed first, and the transaction aborts instead of silently overwriting - the mechanics of deciding exactly when two transactions genuinely collide are worth their own post later.

Where this leaves us

A transaction's view of the database is one number, fixed at creation. Every object it reads is resolved by walking that object's version chain backward until a version old enough to belong to that snapshot turns up. Nothing about this needs a lock - only writers, comparing what they read against what's actually there now, can find out they were too slow. And because that number can be any past version, not just the current one, looking at the database as it stood a moment - or a while - ago isn't a separate feature, it falls out of the same append-only design for free.

The code is on github if you want to read ahead.