Soil: knowing what changed

The transactions post left this on the table with a one-line mention: "'Dirty' itself comes from one of two tracking strategies - either you mark an object dirty yourself, or, as of the current version, Soil detects changes on its own by hashing an object's content and noticing when that hash no longer matches." That's the whole subject of this post: what those two strategies actually are, and how the second one manages to know what changed without ever being told.

The old way was the only way

For most of Soil's life there was exactly one way an object earned the label "dirty": you told it yourself. `transaction markDirty: anObject` was what actually enrolled an object to be serialized on the next commit - skip that call and a real change simply wasn't there as far as commit was concerned. That's still true today under manual tracking; what changed with the current version is that it's no longer the only option, and no longer the default.

This was never a special case bolted onto commit - it's a full strategy object, swappable per transaction. `SoilTransaction` can run under `beManualWriteTracking` or `beAutomaticWriteTracking` (there's a third, `beReadOnly`, for transactions that never write at all), each installing a different `commitStrategy`. Under manual tracking, `markDirty:` does exactly what it always did:

"SoilManualTrackingStrategy"
markDirty: anObject
    transaction addObjectToBeCommited: anObject

`SoilTransaction>>initialize` used to leave that as the only real path. Now it calls `self beAutomaticWriteTracking` by default - manual tracking is a deliberate opt-in switch away, not a removed feature.

The trade-off between the two is straightforward. Manual tracking is cheap - no hashing anywhere, just adding a record to a list when `markDirty:` is actually called - but only as reliable as remembering to call it. Miss one after a real change, and that change is never in the list of things to serialize; nothing errors, it just isn't there at commit. Automatic tracking removes that risk but not for free: every tracked object gets hashed twice - once when it's loaded, to record the starting fingerprint, and once again right before commit, to check whether anything moved.

The check itself

Under automatic tracking, calling `markDirty:` still compiles, but it does nothing:

"SoilAutomaticTrackingStrategy"
markDirty: anObject
    "do nothing"

Instead, Soil figures out on its own which objects changed, at the one point it actually needs to know - right before commit:

"SoilAutomaticTrackingStrategy"
prepareCommit
    "iterate over a snapshot, not the live objectMap: resolving a reference while
    checking #hasChanged can register further objects into objectMap, and growing
    a Dictionary while it is being #do:-iterated can silently skip entries that were
    already there, dropping a changed record from the commit without any error"
    transaction objectMap values do: [ :each |
        each hasChanged ifTrue: [
            self addRecord: each ] ].
    self serializeObjects

That comment is doing real work: resolving a lazy reference while checking one object can pull another object into `objectMap` for the first time, and mutating a Dictionary while iterating it live can silently skip entries. Walking a snapshot instead is what keeps that safe.

`hasChanged` itself is almost nothing:

hasChanged
    ^ fingerprint ~= object soilRecursiveHash

`fingerprint` is a hash taken once, the moment an object is loaded from disk:

ensureFingerprint
    fingerprint ifNil: [
        fingerprint := object soilRecursiveHash ]

So the whole mechanism is: remember a hash when an object comes in, compare it against a fresh hash right before it might go back out. No dirty flags, no proxy setters intercepting writes - just two hashes, taken at two points in time.

Read-only, two ways

There's a third strategy besides manual and automatic: `beReadOnly`, for transactions that are never supposed to write anything. It reuses the exact same fingerprint mechanism from above, just pointed at a different job - catching accidental writes instead of collecting real ones - and it comes in two modes, neither of which is picked automatically: whoever opens the read-only transaction has to choose one explicitly.

The plain, cheap mode is `ignoreWriteAttempts`. With it set, both the fingerprinting and the check disappear:

"SoilReadOnlyStrategy"
recordMaterialized: record materializer: materializer
    super recordMaterialized: record materializer: materializer.
    "if we want to find out what has changed we need to make a
    fingerprint even in read-only mode"
    ignoreWriteAttempts ifFalse: [
        record ensureFingerprint ]

With `ignoreWriteAttempts` true, `ensureFingerprint` never runs, `prepareCommit` never checks anything, and an accidental change is simply dropped rather than raised. No fingerprint gets taken, no comparison gets made - not once, let alone twice like under automatic tracking. That makes it the fast mode: a read transaction pays nothing at all for tracking, which is exactly what a transaction that's read-only simply because that's what the situation calls for should cost. Paying for a hash nobody's going to check would just be wasted work - and worth optimizing for, since in most applications read-only operations vastly outnumber writes.

The other mode, `errorOnWriteAttempts`, opts back into the check instead of skipping it:

"SoilReadOnlyStrategy"
prepareCommit
    ignoreWriteAttempts ifFalse: [
        transaction objectMap values do: [ :each |
            each hasChanged ifTrue: [
                ^ SoilWriteOnReadOnlyTransaction new
                    object: each object;
                    signal ] ] ]

Nothing here ever gets serialized - the moment it finds a change, it signals `SoilWriteOnReadOnlyTransaction` instead of adding a record. That makes it a genuinely useful test tool: wrap a piece of code that's only supposed to read in a transaction set to `errorOnWriteAttempts`, and any write it accidentally makes - a memoizing getter with a bug, an object attached to the wrong place - fails loudly right where it happened, instead of quietly showing up in some unrelated commit later. The cost is exactly the automatic-tracking cost from earlier: a fingerprint taken on materialize, a hash recomputed and compared before commit - paid here purely to prove that nothing changed.

A hash that knows about graphs

`soilRecursiveHash` isn't a generic Pharo hash - it's built specifically so that a change anywhere in an object graph shows up, and so that two structurally different graphs don't accidentally collide:

soilRecursiveHash
    ^ self soilRecursiveHashWith: (IdentitySet new)

soilRecursiveHashWith: aSet
    ^ (self class classLayout soilRecursiveHash: self with: aSet)
        bitXor: aSet size * 16r01000193

The actual walk is delegated to the object's class layout, which knows how that particular kind of object stores its state - named instance variables, indexable slots, or neither. For an ordinary object with named instance variables:

"FixedLayout"
soilRecursiveHash: anObject with: aSet
    (aSet includes: anObject) ifTrue: [ ^ 0 ].
    aSet add: anObject.
    | hash | hash := anObject basicIdentityHash.
    1 to: anObject class instSize do: [ :n |
        hash := hash bitXor: ((anObject instVarAt: n) soilRecursiveHashWith: aSet) * n ].
    ^ hash

Traversing a small object graph to build one recursive hash: each reference contributes its slot's hash weighted by position, and a reference back to an already-visited object short-circuits to 0 instead of looping

Using `basicIdentityHash` as part of that hash only works because of something particular to how Soil uses objects: an object materialized inside a transaction is private to that transaction. It's never shared with another transaction's own materialized copy of the "same" persistent object, and it only ever gets compared against a fingerprint taken from itself, earlier, in the same transaction. There's no case where two different object instances representing the same logical value need to hash the same - `hasChanged` always asks "did this exact object change since it was loaded," never "does this object match some other one." That's exactly the situation where mixing in an object's own identity is safe rather than fragile.

Two things here are deliberate, not incidental. The `IdentitySet` check is what keeps a cyclic graph from recursing forever - once an object has been visited, revisiting it contributes nothing (`^ 0`) instead of looping. And every instance variable's hash gets multiplied by its own slot position (`* n`) before being folded in. `bitXor:` alone is commutative, so without that weighting, two swapped values would hash identically to their unswapped selves. Real tests prove this matters: swap the two elements of `Array with: #foo with: #bar` and the hash changes; swap two keys in a Dictionary and it changes too - exactly the kind of change a plain, unweighted XOR would silently miss.

Two special cases

Immediate values - SmallIntegers, Characters, and the like - don't have instance variables to walk, so they hash to their raw identity hash. That creates its own trap: `65` and `$A` are, bit for bit, the same value, just interpreted differently. A real test - named, in the source, "the immediate mapping problem" - checks exactly this: `65 soilRecursiveHash` and `$A soilRecursiveHash` must not collide, or a change from one to the other could go undetected.

Proxies get their own override, and for a different reason. A `SoilObjectProxy` stands in for an object that hasn't been loaded from disk yet, so hashing it the normal way would force exactly the load it exists to avoid:

"SoilObjectProxy"
soilRecursiveHashWith: aSet
    "proxies are identified by their object id. If proxies are exchanged the
    hash should stay the same as it refers to the same object"
    ^ objectId soilRecursiveHashWith: aSet

A proxy hashes through its `objectId` instead of resolving what it points to. Computing a fingerprint over a graph with unresolved branches never pulls those branches into memory just to find out if they changed.

The read that looks like a write

Because the hash walks every instance variable it finds, anything that writes into one as a side effect of being read looks exactly like a real change. The obvious case is a memoizing getter - `^ cache ifNil: [ cache := self computeCache ]` - where simply reading a value the first time quietly mutates the object that holds it. Under automatic tracking, that's indistinguishable from an actual domain change: the fingerprint taken at load time won't match anymore, and the object gets serialized on the next commit for no reason a caller would recognize as "I changed something."

The escape hatch is `soilTransientInstVars`: instance variables named there are skipped both when an object is serialized and when it's hashed, so whatever they hold never counts toward `hasChanged`. Soil relies on this for its own machinery, not just as a theoretical option - `SoilIndexedDictionary`, which backs every indexed collection, marks its entire runtime wiring transient:

"SoilIndexedDictionary class"
soilTransientInstVars
    ^ #( index segment dirty transaction )

All four of those get rebuilt every time the object is materialized, so none of them should ever make the object look changed on their own.

Where this leaves us

Automatic tracking didn't remove manual tracking, it just stopped being the thing you have to reach for by default: `markDirty:` still exists, still works exactly as before, one `beManualWriteTracking` call away. What runs instead by default is two hashes and a comparison - a recursive, graph-aware, cycle-safe hash taken once when an object is loaded, and the same hash taken again right before commit. The two special cases for immediates and proxies, and the transient-instance-variable escape hatch, all exist to keep that comparison honest: catching real changes, without forcing loads it doesn't need or flagging changes that were never really there.

The code is on github if you want to read ahead.