This post is part of an ongoing series building Soil, a file-based, transactional object database for Pharo, from the ground up — one real mechanism at a time.
Soil: checkpoints and recovering from a crash
The transactions post left this dangling: "if any of this actually changed the database, Soil checkpoints afterward - flushing everything to disk in a way that can survive a crash. What a checkpoint really guarantees is its own post." The journal post added its own loose end on top: every fragment file has to start with a checkpoint entry, for reasons that would only matter once recovery needed them. Both loose ends belong to the same story.
The reason any of this matters is that a crash can land at any moment - the Pharo image can die, the disk underneath it can fail, the whole machine can lose power - and none of that is under Soil's control. Protecting against it forces a strict order on things: a journal entry has to be durable before anything is allowed to depend on it, the corresponding change actually lands in the heap next, and only once that heap write has itself been flushed to disk does a checkpoint get to mark the result as a new trusted point. What a crash does to each of those phases, taken one at a time, is most of what the rest of this post works through.
A checkpoint is the one moment where two very different things briefly line up: a position in the journal, and the actual state of the heap and control file on disk. Before that moment, the heap can be ahead of what's been flushed - written in memory or into an OS buffer, but not yet guaranteed to survive a crash. After it, everything up to that journal position is durably on disk, in both places at once, and that's exactly the position recovery starts from.
What a checkpoint actually does
A checkpoint is short, and every step in it is deliberate:
"Soil"
checkpoint
| entry checkpointLSN |
semaphore critical: [
entry := SoilCheckpointEntry new
previousCheckpoint: self control lastCheckpoint.
checkpointLSN := self journal writeEntry: entry.
self control checkpoint: checkpointLSN.
entry commitIn: self ].
^ entry
First, a new checkpoint entry is built, and it points at the previous checkpoint's own position - checkpoints form a chain, the same way object versions did back in the MVCC post, just one level up: instead of chaining an object's history, this chains the database's own history of "known-good" points. That entry gets written into the journal like any other entry, which is what gives it its own LSN. Only then does the control file get told this is now the last checkpoint. And only after all of that does the actual, expensive part happen: entry commitIn: self flushes the behavior registry, the object repository, and the control file, in that order, syncing each one for real if isFsyncEnabled says so - the same flush-versus-fsync split the journal post already showed for individual entries, just run once across everything at the end instead of once per append.
The control file's one important number
SoilControlFile is a small, fixed binary file: a database format version, a database version, an application version, and one more field that matters here - the position of the last checkpoint, stored as a plain LSN:
"SoilControlFile"
lastCheckpoint
^ semaphore critical: [
self stream
position: self checkpointPosition.
(stream next: self checkpointPositionSize) asInteger asLogSequenceNumber ]
checkpoint: anInteger
semaphore critical: [
self stream
position: self checkpointPosition;
nextPutAll: (anInteger value asByteArrayOfSize: self checkpointPositionSize);
flush ]
That's it - one number, in one fixed place, that answers exactly one question on startup: which fragment file, and which byte inside it, is the last point Soil is sure was fully and safely written.
A number Postgres would recognize
That number is a SoilLogSequenceNumber - log sequence number, a term borrowed directly from Postgres, which uses it for exactly the same purpose in its own WAL: a single, monotonically growing address into an append-only log. Soil's version packs two things into one integer:
fileNumber
"the high 40 bits are making up the file number"
^ value bitShift: -24
fileOffset
^ value bitAnd: 16rFFFFFF
Which fragment file, and which byte inside it - the same idea the object index already gave the heap, a stable address that turns "where is this" into a direct lookup instead of a scan. The 24 bits set aside for the offset aren't arbitrary: they're exactly what a fragment file's 16-MB size limit fits into, and the fragment filenames themselves are just that same file number, hex-padded:
"SoilJournalFragmentFile class"
filenameFrom: anInteger
^ (anInteger printStringBase: 16 length: 10 padded: true) asLowercase
Fixed-size, sequentially-numbered files are also what makes it possible to ever get rid of old ones - and a checkpoint is exactly the fact that would make deleting one safe: once a checkpoint's LSN is at or past the end of a fragment file, recovery will never need to open that file again. Soil doesn't actually check that before deleting, though. It just keeps at most a configured number of the newest fragment files, and creating a new one deletes whatever's oldest by count:
"SoilPersistentDatabaseJournal"
createFragmentFile: filename
self removeFragmentFiles.
(self path / filename) ensureCreateFile.
^ (self openFragmentFile: filename)
initializeFilesystem;
yourself
removeFragmentFiles
| files |
maxFragmentFiles ifNotNil: [
((files := self sortedFiles) size >= maxFragmentFiles) ifTrue: [
(files allButFirst: maxFragmentFiles - 1) do: #delete ] ]
That's the same problem Postgres solves by recycling its own 16-MB WAL segments instead of ever letting one file grow forever - space can only be reclaimed, or history eventually archived and rotated, if the log is chopped into fixed-size pieces to begin with. Soil's version is blunter than Postgres's: it deletes outright rather than renaming a file for reuse, and by count rather than by checking what a checkpoint has actually made safe to discard - but the reason the size is capped at all is the same one.
Picking up where the last checkpoint left off
Recovery starts from that one number:
"SoilDatabaseRecovery"
recover
| lastCheckpoint checkpointEntry fragmentFile |
lastCheckpoint := soil control lastCheckpoint.
fragmentFile := journal openFragmentForLSN: lastCheckpoint.
checkpointEntry := SoilJournalEntry readFrom: fragmentFile stream.
fragmentFile atEnd ifTrue: [ ^ self ].
self readFragmentFileProtected: fragmentFile.
fragmentFile close.
lastCheckpoint fileNumber + 1 to: journal lastFileNumber do: [ :fileNumber |
fragmentFile := journal openFragmentFileNumber: fileNumber.
fragmentFile setToStart.
self readFragmentFileProtected: fragmentFile.
fragmentFile close ].
soil checkpoint
Open the fragment file the last checkpoint points to, read the checkpoint entry itself - if that's the last thing in the file, there's nothing left to redo, and recovery is done before it really started. If there's more after it, everything from there onward gets replayed: a begin-transaction entry starts collecting a fresh SoilTransactionJournal, entries get added until a commit entry closes it, and then it's applied with transactionJournal commitIn: soil recovery: true - the exact same method every entry already implements for an ordinary commit. Recovery isn't a separate apply mechanism sitting next to normal commit; it's normal commit, run again, later, on entries that turn out to have already made it to disk but maybe not to the heap. Any further fragment files after the checkpoint's own get replayed the same way, in order, since a single commit can span more than one file.
This is exactly the payoff of every entry's commitIn:recovery: only ever needing the database itself, never the transaction that produced it - the reason the journal post insisted on that in the first place. Recovery runs from Soil>>open, before a single transaction exists in this process, quite possibly not even the same process that wrote the entries down. A journal entry that needed to ask its original transaction anything couldn't be replayed here at all; one that only needs soil doesn't notice the difference between committing live and being replayed hours later.
When the tape cuts off mid-entry
A crash can land in the middle of writing an entry, not just between them. Reading a half-written entry raises MessageNotUnderstood on whatever byte comes next, and recovery catches exactly that and turns it into SoilTruncatedRead. It only tolerates that in one specific place - the very last fragment file:
"SoilDatabaseRecovery"
readFragmentFileProtected: aSoilFragmentFile
[ self readFragmentFile: aSoilFragmentFile ]
on: SoilTruncatedRead
do: [:err |
(journal lastFileNumber = aSoilFragmentFile fileNumber)
ifFalse: [ SoilDatabaseIsInconsistent signal: 'after a truncated file there should not be another one' ].
journal
currentFragmentFile: aSoilFragmentFile;
cycleFragmentFile ]
A truncated read anywhere but the last file means something is wrong in a way recovery isn't willing to guess its way past - that raises SoilDatabaseIsInconsistent instead of quietly continuing. A truncated read in the last file is exactly what a crash mid-write looks like, and the fix is blunt: whatever was being written is simply gone, and the journal moves on to a fresh fragment file from here. That's a real, named trade-off, not a hidden one - the entry that didn't finish writing never existed as far as the database is concerned, and nothing tries to reconstruct it.
A crash doesn't have to cut a byte in half to leave something incomplete, though. It can just as easily land cleanly between two entries - each one reads back perfectly, nothing corrupted - but stop before the transaction's closing commit entry was ever written. Recovery catches that too, because reading a transaction's journal is written to expect a commit entry as the price of admission:
"SoilDatabaseRecovery"
readTransactionJournal: transactionJournal from: stream
| entry |
[ stream atEnd ] whileFalse: [
entry := self readEntryFrom: stream.
transactionJournal addEntry: entry.
(entry class == SoilCommitTransactionEntry)
ifTrue: [ ^ transactionJournal ] ].
SoilTruncatedRead signal: 'reading of transaction journal is incomplete'
Running out of stream before that ifTrue: fires raises the exact same SoilTruncatedRead, handled by the exact same protected read shown above - discarded if it's the last file, an inconsistency if it isn't. A transaction only counts as having happened once its commit entry is there to be read back; a string of otherwise-perfect entries missing just that one is worth exactly as much as no entries at all. Once replay is done, recover writes a brand new checkpoint, so the next crash - if there is one - has less to redo.
A crash at every phase
With that mechanism in place, it's worth walking through commit and checkpoint step by step and asking, for each one: what if the crash lands exactly here?
Most of commit is easy. Everything before writeTransactionJournal: - building the journal, serializing dirty objects - only exists in memory. A crash there leaves nothing behind to even notice; the transaction just never happened. A crash during writeTransactionJournal: itself is the previous section's story, byte-truncated or missing its commit entry either way. The interesting case is the gap right after: the commit entry has made it to disk, writeTransactionJournal: has returned, but journal commitIn: soil - the step that actually writes the new object versions and index updates - hasn't finished, or hasn't even started yet. A crash there doesn't lose anything, because replay doesn't know or care that this ever happened live: it finds the same commit entry sitting durably in the journal and runs transactionJournal commitIn: soil recovery: true on it - the identical method a live commit would have called. Whether that write happens once, during the original commit, or again later, during recovery, makes no difference to the result. That's the entire point of a write-ahead log: the log is what's durable first, and everything downstream of it can always be reproduced from it, however late.
A checkpoint follows immediately after every commit that changed anything, so the same walk continues right on from there. A crash before the checkpoint entry is even written just leaves the old checkpoint standing - the next recovery starts from there and replays a bit more than strictly necessary, including the commit that just happened, which is safe for the reason above. A crash while the checkpoint entry itself is being written is, again, the previous section's mechanism: a truncated entry, discarded.
The remaining two gaps are checkpoint-specific, and recovery handles each by name. The checkpoint entry can be fully written to the journal before a crash hits, with the control file not yet updated to point at it:
"SoilDatabaseRecovery"
"reading a checkpoint entry is unlikely. It should be possible only if there was an
error after writing the checkpoint entry and before marking that position in the control
file. But it is possible it happens so we just need to complete it by updating the
checkpoint position in the control file"
(entry class = SoilCheckpointEntry) ifTrue: [
soil control checkpoint: lsn ]
Recovery starts from the control file's old, still-valid position, replays forward, and runs straight into the checkpoint entry it's already sitting on top of - at which point it just finishes the one thing that didn't happen before the crash, marking the control file with that position. Nothing gets replayed twice, and nothing gets lost; the crash only meant one bookkeeping step ran a little later than planned.
The last gap is the one the checkpoint's own write order exists to close. writeEverythingToDisk flushes and syncs the behavior registry, then the object repository, then the control file - in that order, every time. That order is not incidental: the control file's own sync is the one action that makes the new checkpoint position physically durable, and it only runs after everything that position vouches for has already been synced. A crash anywhere before the control file's own sync completes can only mean the control file on disk still shows the previous checkpoint - never a new one pointing at data that isn't actually there yet. Worst case, recovery redoes a bit more work than it strictly needed to. It can never redo too little.
Where this leaves us
A checkpoint is a chain of its own, each one pointing at the last, each one only trusted once the entry describing it is durably in the journal and everything else has actually been flushed and synced to disk. Recovery does nothing clever with that: read the last trusted position, replay whatever comes after it using the same commit logic that put it there in the first place, and treat anything left unfinished by a crash as gone rather than worth guessing at. Walk through commit and checkpoint at whatever granularity you like, and the same pattern keeps repeating: at any given instant, something is either not durable yet and safe to lose, or already durable and safe to redo. There's no third case, and no instant that falls outside both. The database doesn't try to be smarter than the journal it's built on.
The code is on github if you want to read ahead.