Soil v6 released

Soil v6 is out. As with v5, that means a new branch v6 you can load, not a tag. No migration is needed if you are coming from v5: the on-disk format is unchanged, and journal fragments written by v5 are still read.

This release is the first one that came with a blog series running alongside it. Several of the bigger topics have a post of their own, so this one only says what is in the release and points to where it is explained.

Windows. Soil runs on Windows now. File locking and flushing go through the platform, every file handle is closed explicitly, and the test suite runs on Windows in CI. This builds on the original Windows work by @mumez and @riverdusty, thank you both. Pharo 14 is a supported load target as well.

Residents. The new package Soil-Resident is the biggest addition, and the one that changes what Soil is. Until now Soil was a database that only reacts. With residents, processes live in the database: a persisted description, a transient instance created from it when the database opens, and a supervisor that starts, watches, restarts and stops them. Detached objects let a resident hold an object across transactions. How and why is the subject of Resident Eval - making persistence undead.

Consistency check and repair. Soil>>checkConsistency stopped at the first violation. It is replaced by trackers that walk the database once and report findings, each finding knowing the repairs that fit it: recompute an index size, rebuild skip list links, cut a broken version chain and a few more. The check runs on a live database at the version of its transaction, and a Roassal view draws the object graph with the findings attached.

Indexes. There are persistent cursors, positions on an index that outlive their transaction, scans bound to a key prefix, and a move that re-keys an entry without materializing its value. Concurrent removal of the same key is now a commit conflict (SoilIndexHasConcurrentChange) instead of a KeyNotFound, and index conflicts are detected from the journals alone. The two posts on indexes on disk and indexes under MVCC describe the machinery these changes live in.

Backup, read-only and recovery. A backup copies indexes at the version its records are read at and leaves the source journal out unless asked for it. Soil>>readOnly opens a whole database read-only, a reader opens past a journal that is not checkpointed yet, and recovery says which way it went, or refuses with a message when it is needed but cannot be written. The background is in the posts on the journal and on checkpoints and recovery.

Export, import and migration. The graph importer understands the version chain of a class, so importing into a database that holds another shape of it no longer fails. Application migrations run step by step, each version in a transaction of its own, and report their progress.

Performance. Commits and imports write their records in one go, a commit finds its records by hash instead of a linear search, and writing an object no longer reads the clusters it points at.

A few changes need attention when you upgrade. checkConsistency is gone, as described above. Transactions now ignore write attempts by default, raising on them is a deliberate choice (errorOnWriteAttempts). And the file lock classes got the Soil prefix (SoilMacOSFileLock, SoilUnixFileLock).

The v5 post named three things for v6. Windows support made it. The online garbage collector did not, but v6 adds more groundwork for it, such as SoilRecordLocation and a metric for how far the oldest reader lags. Process-specific variables did not make it either. Next on the list are multi-index collections and partial indexes, the online garbage collector, copy-on-write for index pages so that a read-only transaction copies nothing, and making Windows support official.

Thanks to everybody who contributed to this release: Marcus Denker, Andrzej Prochyra, Paul Wilke, Astares, @mumez and @riverdusty.

The full changelog is here, the announcement Marcus and I wrote for the repo is here, and the code as always is on github.

Tags: Soil, Persistence, Oss
Found something worth flagging? Send feedback.
This work is licensed under CC BY 4.0.