This post is part of an ongoing series building Soil, a file-based, transactional object database for Pharo, from the ground up — one real mechanism at a time.
Soil: the journal, and what makes it a WAL
The transactions post left this one dangling: "Every new or changed object, every index update, and the new database version becomes an entry in a journal - an in-memory list first, not yet on disk [...] The journal is central enough to deserve a full post of its own later." This is that post.
From transaction to journal entries
The thing a transaction assembles is a SoilTransactionJournal - one object per transaction, an ordered list of SoilJournalEntrys. It doesn't wait for commit to come into existence: the moment anything asks the transaction for its journal at all, one gets created with a begin-transaction entry already sitting in it, timestamped and stamped with the transaction's read version. Everything that happens afterward just adds to that same list.
What gets added, and when, splits into two routes - and the reason for the split is exactly the mechanism the modification-tracking post already covered. That post showed SoilIndexedDictionary marking its own index, segment, dirty and transaction instance variables transient, specifically so the recursive hash never looks at them. A change to an index's actual structure is therefore invisible to automatic tracking by design - no fingerprint comparison at commit time will ever notice it happened. There's no "later" for it to be picked up at, so it has to be logged the instant it happens, right inside the operation making the change:
"SoilIndexedDictionary"
at: key put: anObject
| objectId oldValue |
objectId := transaction makeRoot: anObject.
oldValue := self newIterator at: key add: objectId.
transaction addJournalEntry: (SoilAddKeyEntry new
segment: [ self segment ];
id: id;
key: (index indexKey: key);
value: objectId;
oldValue: oldValue).
self indexUpdated.
^ objectId
Ordinary object changes don't have that problem - they're exactly what the recursive hash does see - so there's no rush: those only get turned into journal entries once, at commit, from whatever tracking found dirty. Both kinds end up as the same sort of thing once they're in the journal: a small, self-describing record with a type tag out front, the same shape the very first posts in this series used for individual objects, just one level up - instead of tagging a value's type, it tags what kind of change happened.
Thrown away on abort, written on commit
For as long as a transaction stays open, this journal is just an ordinary object sitting in memory, growing as index entries trickle in - nothing about it is durable, and nothing needs to be yet. If the transaction aborts, that's the whole story: its tracking state gets reset, it's dropped from the transaction manager, and the journal built up so far never gets handed to anything else. It simply becomes garbage along with the rest of the abandoned transaction. There's no undo step to run, because nothing was ever written down anywhere durable that would need undoing in the first place.
Committing is what finally does something with it. First, outside any lock, every dirty object gets serialized into bytes - the step the modification-tracking post walked through in detail. Only then, inside Soil's single global critical section, does the transaction fill in everything still missing from its journal: one entry per new object version, the bumps to each touched segment's index, the new database version, and a closing commit entry. That's the same journal the index entries had already been landing in throughout the transaction - commit doesn't start a new list, it finishes the one that was already there. Only once all of that is in place does the transaction hand the whole thing over with writeTransactionJournal: - and that's the exact moment this stops being one transaction's private, in-memory list and starts being part of the database's own journal.
Logical operations, not physical bytes
It would be easy to assume a journal entry records something like "write these exact bytes at this exact file offset" - a physical, page-level WAL, the kind that just replays raw patches without knowing or caring what they mean. That's not what these entries do. Each one describes an operation, and knows how to actually perform it:
"SoilNewObjectEntry"
commitIn: soil recovery: aBoolean
position := soil objectRepository
at: objectId
putBytes: bytes.
record ifNotNil: [
record position: position.
record := nil ]
The entry doesn't carry a target position at all - it carries an object id and a payload, and putBytes: decides where that payload actually lands, whether this is running as part of an ordinary commit or being replayed during recovery. The position only exists as an outcome of applying the entry, never as an input to it. A segment-index entry works the same way, just one level up: it says "segment 3's last index is now 517," never "byte 12 at offset 900 becomes 0x02." Applying an entry always means running a small piece of real logic against the database, not copying bytes into place.
That's exactly what lets the same entries serve two very different purposes through the same method, commitIn:recovery:: driving an ordinary commit the first time, and replaying history during recovery, later, possibly in a different process entirely. Look again at that signature, though - commitIn: soil recovery: aBoolean, not commitIn: aTransaction. Every entry only ever needs the database itself to apply itself; nothing in it reaches back into the transaction that originally produced it, asks it a question, or depends on it still being around. That's what makes recovery possible at all. Recovery runs from Soil>>open, at the very moment a database is opened - before a single transaction has been created, sometimes before anything else has even started. There is no transaction for a replayed entry to lean on at that point, so it's a good thing none of them ever needed one.
Some entries remember what they replaced
A type tag and a payload undersells one detail: some entries also carry the value they overwrote. SoilAddKeyEntry and SoilRemoveKeyEntry both have an oldValue field, and it isn't there for tidiness.
Once a key in an index changes, the index's own on-disk page has moved forward - and unlike the heap, which keeps every old version around untouched via previousVersionPosition (the whole subject of the MVCC post), an index page gets mutated in place, copy-on-write per transaction. There's no version chain sitting on disk for an older reader to walk back through. What there is instead is the journal itself: if a reader's page has been touched by transactions newer than that reader's own snapshot, Soil walks backward through those transactions' journal entries looking for the one that changed this exact key, and hands back its oldValue in place of the current one. It's the same idea as the heap's version chain - give an old reader back what used to be there - just built by replaying journal history on demand instead of following a chain of on-disk positions.
Objects have something that looks like the same idea, and it's worth being honest about what it's actually for. Whenever a new version of an existing object gets committed, the entry for it captures the entire previous version's serialized bytes into its own oldBytes field, right as that entry gets built:
"SoilNewClusterVersion"
asJournalEntry
| objectEntry entries |
entries := OrderedCollection new.
objectEntry := self hasPreviousVersion
ifTrue: [
SoilNewObjectVersionEntry new
oldBytes: previousVersionRecord serialize ]
ifFalse: [ SoilNewObjectEntry new ].
objectEntry
record: self;
objectId: objectId;
bytes: self serialize.
entries add: objectEntry.
^ entries
That's real data, captured at real cost, every time an object gets a new version. Unlike oldValue on an index entry, nothing reads oldBytes back today - objects already get their MVCC for free from the heap's own version chain, so there's no reader that needs it for that particular job. It's there for a different reason: a possible rollback, undoing a committed version back to exactly what it replaced, straight from the journal, without touching the heap's own version chain at all. That capability just hasn't been built on top of the field yet.
Two journals, one interface
SoilDatabaseJournal is only the abstract base, and its two subclasses aren't two flavors of the same guarantee - only one of them can survive a crash. SoilMemoryDatabaseJournal holds transaction journals in a plain in-memory Dictionary and nothing else: no fragment files, no fsync, nothing that outlives the image - a journal in name and shape only. SoilPersistentDatabaseJournal is the one that actually earns the term write-ahead log: it writes to disk, in fragment files, with checkpoints and recovery behind it. Both answer the same questions - which transaction journals exist, in what order - so the rest of Soil never has to care which one it's talking to. Everything from here on is about what the persistent one actually does with that content once it has to survive a restart.
Where the bytes actually land
For the persistent journal, entries end up in a SoilJournalFragmentFile, and every fragment file has a hard, fixed ceiling of 16 MB - a plain size check against the stream, nothing more elaborate. Once a file is full, cycleFragmentFile flushes and closes it and opens the next one, numbered sequentially. Every fragment file is also required to start with a checkpoint entry. Neither of those two facts - the fixed size, or the checkpoint-first rule - earns its keep here; both only really pay off once recovery has to decide where to start reading after a crash. That's also where a name worth knowing shows up: every position in this whole structure gets its own single-number address, a log sequence number, a term and an idea borrowed straight from Postgres - and how fragment files get numbered, rotated, and eventually deleted around that address is exactly where the next post picks up.
flush versus fsync
Appending an entry takes a lock, seeks to the end of the current fragment file, writes the entry, and always flushes before releasing that lock again. stream flush only pushes bytes out of the Smalltalk-level buffer and into the operating system - it says nothing about whether they've actually reached the disk platter or SSD cell rather than sitting in an OS write-back cache a power loss could still erase. Reaching the disk for real is a separate, explicitly optional step, gated behind a setting:
writeContentsToDisk
databaseJournal soil setup isFsyncEnabled ifTrue: [
stream sync ]
stream sync is the real fsync, and it only runs at all if isFsyncEnabled says so. The reason fsync matters is the same reason it's slow: forcing the operating system to actually push bytes to the physical device, instead of trusting its own write-back cache, is what turns "written" into a guarantee - and a guarantee like that costs real time. Running a large test suite with it switched on can be something like twenty times slower than with it off, and that's exactly the situation isFsyncEnabled exists for: a test doesn't need to survive a real crash, so there's nothing lost by turning the guarantee off there. Normal operation is a different story. Both committing and checkpointing are built on the assumption that once they've happened, the result is actually durable - switching fsync off breaks exactly that assumption, quietly, which is why it isn't something to reach for outside a case like CI.
Where this leaves us
A transaction's journal starts life as an ordinary in-memory object, filled in two ways for two different reasons: index changes the instant they happen, because tracking would never otherwise see them; object changes only at commit, because tracking already did. Abort the transaction, and that's all the journal ever was. Commit it, and it gets handed to whichever database journal is actually backing this Soil - a real write-ahead log if it's SoilPersistentDatabaseJournal, fixed-size fragment files synced to disk on demand rather than by accident; a plain in-memory stand-in if it's SoilMemoryDatabaseJournal, gone the moment the image is. Either way, what makes an entry replayable - a logical operation that only needs the database itself to apply, never the transaction that produced it - stays exactly the same. What happens when the database restarts and has to read all of this back in is next.
The code is on github if you want to read ahead.