Soil: transactions and how a commit happens
The last post gave every object a stable id and a way to find its current bytes through the index. That's enough to read and write single objects. What's still missing is the thing that makes a database out of that: a way to change several objects at once and have it count as one atomic step. That's a transaction, and this post is about what one actually is in Soil and what happens when you commit it.
A transaction is just an object
In a lot of systems, "begin a transaction" reaches into some ambient, thread-local state: the connection you're holding remembers that you're "in a transaction" until you commit or roll back. Soil doesn't have that. A transaction is an ordinary object you get back from `soil newTransaction`, and it's yours to keep, pass to other methods, or hold onto for a while:
txn := soil newTransaction.
txn root: 5@2.
txn commit.Nothing here is tied to the process running it - not yet, anyway. One process can hold several transactions open at the same time, simply because it's holding several transaction objects; there's no single "current transaction" a process is limited to. Right now that means threading the transaction through your own code by hand, since Soil won't find it for you. That's on its way to changing, though: process-specific variables are planned for the next version, which would let a process bind a transaction to itself implicitly instead of always passing it around explicitly.
The root: the one address nobody has to look up
Every Soil database has exactly one object with a known location: the root, at the fixed id we saw in the last post, segment 1, index 1. `txn root: anObject` makes that object (and everything it transitively points to) the graph the transaction manages; `txn root` reads it back. Anything reachable from there can be found; anything that isn't reachable from the root, directly or indirectly, might as well not exist as far as the database is concerned.
Commit, step by step
Calling `commit` hands the transaction to Soil's transaction manager, which does two things: run the actual commit, then checkpoint if anything changed. The commit itself splits cleanly into a cheap part and an expensive part.
The cheap part happens before any locks are taken: every dirty object gets serialized into bytes. "Dirty" itself comes from one of two tracking strategies - either you mark an object dirty yourself, or, as of the current version, Soil detects changes on its own by hashing an object's content and noticing when that hash no longer matches. Which strategy is in play doesn't change what happens here; it only changes how an object earns the label "dirty" in the first place, and that's a topic detailed enough to deserve its own post later. Either way, serialization doesn't need exclusive access to anything, so it happens outside of any locking, and if there's nothing to commit, the transaction can bail out here without touching shared state at all.
The expensive part is wrapped in a single global critical section - only one transaction across the whole database can be in here at a time. Inside it, in order:
- Lock the database version and bump it. The database version is a single, strictly increasing number that versions the entire state of the database, not just one object - every object this transaction writes gets stamped with the same new version. Reading it and incrementing it is the moment the transaction stakes its claim: whatever number it reads becomes the write version for everything it's about to commit.
- Lock every record and check for conflicts. For each object the transaction is about to write, Soil locks it and compares the version it was read at against the version currently on disk. If someone else committed a change to the same object in the meantime, this is where it's caught: Soil raises an exception and the transaction aborts instead of committing - the mechanics of that check are worth their own post later.
- Lock the tail of every touched segment. New objects need a fresh index slot, so the last-used index number of each segment involved gets locked too, to hand out index numbers safely. This is also where those placeholder ids with index 0 from the last post stop being placeholders: each one gets assigned the next available index in its segment.
- Build the journal. Every new or changed object, every index update, and the new database version becomes an entry in a journal - an in-memory list first, not yet on disk.
- Validate, then write the journal to disk. This step is doing more than it sounds like. The journal takes every change this transaction made and abstracts it away from the transaction itself, into standalone modification entries that stand on their own. Those entries get appended to Soil's journal - an append-only file, never overwritten - and then handed to `fsync()` (on Unix systems), which is what actually guarantees they've reached disk rather than sitting in a buffer a crash could still wipe out. Only after that does Soil apply the entries: new object bytes land in the heap, index slots get repointed, segment tail counters move. The journal is central enough to deserve a full post of its own later.
- Release every lock, whether the commit succeeded or raised partway through - this step runs in an `ensure:` block, so a failed commit can't leave the database half-locked.
If any of this actually changed the database, Soil checkpoints afterward - flushing everything to disk in a way that can survive a crash. What a checkpoint really guarantees is its own post.
One writer at a time
That global critical section is a deliberate, simple choice: rather than fine-grained locking across the whole database, Soil serializes the expensive part of every commit through one single lock. It's the same trade-off SQLite makes with its single-writer model - happy to let many readers proceed concurrently, but only one writer gets to commit at a time. For Soil, that's acceptable because the expensive part - locking records, writing the journal - is meant to be short; the genuinely slow part, serialization, already happened before the lock was taken.
Where this leaves us
A transaction is a plain object, created explicitly and passed around explicitly, with one well-known root as its single fixed entry point. Committing it serializes changes outside of any lock, then takes a single global lock to bump the database version, check for conflicts, write a journal, and apply it - releasing the lock unconditionally afterward.
A few things got only a one-line mention here and deserve a full post of their own: how an object earns the label "dirty" in the first place, how a read transaction gets a consistent view of the database without taking any locks at all, which is what MVCC gives us next, and exactly how the conflict check in step 2 decides that two transactions really did collide.
The code is on github if you want to read ahead.