This post is part of an ongoing series building Soil, a file-based, transactional object database for Pharo, from the ground up — one real mechanism at a time.

Read the whole Soil series →

Soil: Resident Eval - making persistence undead

Over the years I have said and written a bit about databases: why we need them at all, how they work inside, and why objects fit so badly into most of them. An object has identity, behavior and references to other objects. Relational databases are the best-known example of a different model: they know tables with rows, columns and foreign keys. Everything between the object and the table has to be translated, and the translation does not stay in the persistence layer. It leaks into the model. Classes grow fields that exist only for the mapping, relationships get shaped like join tables, and a design that started as a network of collaborating objects ends up as a set of records with methods attached. Object-oriented development suffers from this quietly and for a long time, because nobody can point to the moment where it went wrong.

That is one axis of the problem, the impedance mismatch between objects and relations, and Soil removes it by storing objects as objects. There is a second axis next to it that bothers me just as much, and it is a different one. Databases are passive.

We use almost every database the same way. The application opens a socket, sends a query, gets a result, and the database waits until the next query arrives. Whatever the database does, it does because somebody asked, and only for as long as it takes to answer. On the client side a transactional database adds a second restriction: the objects that come back live only within the transaction that read them. When it ends, they are snapshots of something that has moved on.

Both restrictions are tolerable until they are not. Sometimes an object should act across transactions: a tracker that collects changes over minutes, a scheduler that has to remember when it ran last. Sometimes the database itself should take over a task and carry it out in the background, without the application having to build an infrastructure for it first. Who holds my process? Who starts it when the database opens and stops it before it closes? Who notices when it dies, and who brings it back? Every application that needs this builds the same answers again, each a little different and each one outside the database it works on.

Put together, this says that what an application often needs is less a passive database than a runtime: a place where data and the things that work on it live together, with somebody responsible for starting, supervising and stopping them. Databases have always had a bit of this, in triggers, stored procedures and job schedulers, but as add-ons in a language foreign to the application. In Soil the application and the database speak the same language, so the runtime can be made of the same objects.

Soil answers the questions above once, in the package Soil-Resident. The things it manages are called residents. The name is deliberate: a resident is present in the database, what it does is a different matter. One rule runs through the whole design: Soil asks, the embedding application decides. When to start, when to stop, how long to wait for a stop, how often to check on the residents, all of that is passed in by whoever opens the database.

A concrete case: limits

Every resource of a reliable service has to be limited: requests per minute, rows per account, storage per customer. That reaches into the application model, because the limit and the amount used so far are data of the application. The usual solution is outside of it. The service logs every action somewhere, a job aggregates the log in intervals and compares the sums with a quota that is stored somewhere else. It works, and it needs a log, a job, a place for the quotas and a way to keep them in step.

What if that should happen live, without an external service? Then the application needs an object that counts, and the object has to survive a restart, so it has to be persistent. The obvious way is to update the counter in every transaction that uses the resource. That puts the cost of the limit into every single action, and it is worse than that: it creates a conflict point. Every transaction writes the same object, so concurrent transactions conflict on it, and the more traffic there is, the more of them have to be repeated.

The counter does not have to be persistent all the time. An object in memory can count, at the cost of an increment and without any transaction, and be written to the database now and then, in a transaction of its own. A crash then loses at most the counts since the last write, which for a limit is a bounded and usually acceptable error. What such an object needs is a place to live, something that creates it when the database opens, writes it out now and then, writes it a last time when the database closes and notices when it died. That is a resident, and the limits counter is the example for the rest of this post.

Description and instance

A resident has two halves. The SoilResidentDescription is persistent: a name, a reference to a workspace, a flag disabled, and whatever configuration the resident needs. The SoilResident is transient and lives only in the image that runs it. The description creates its instance itself with newInstanceIn:, so there is no table that maps descriptions to classes.

The descriptions are collected in a SoilResidentRegistry, a cluster root of its own that maps a name to a description. The workspace is the place where a resident keeps its state, any object that is a root as well; for the limits resident it is the limits object itself.

Description and instance

The dashed line in the picture is the important part. A materialized Soil object is a different Smalltalk object in every transaction, and a resident outlives many transactions. It therefore never holds the description or the workspace, only their ids, and resolves them inside the transaction it currently runs in: descriptionIn: and workspaceIn:. Nothing of a resident is persisted, its instance variables are all transient. There is one deliberate exception to holding only ids, the detached object, and the limits resident needs it.

Active residents work in steps

A SoilActiveResident owns a process, and that process runs a loop. One turn of the loop is one step, and one step is one transaction:

runLoop
	[ self shouldStop ] whileFalse: [
		self runStrategy awaitDue.
		self shouldStop ifFalse: [ self runStepProtected ] ].
	self markStopped

What awaitDue waits for is decided by the run strategy, and these are the three that exist:

Strategy Runs a step Typical use
SoilSignalRunStrategy when somebody signals that there is work react to changes
SoilIntervalRunStrategy every interval, the first one immediately (default one minute) periodic work
SoilCoalescingRunStrategy once per window after the first signal, however many signals came in (default 30 seconds) batch up bursts

An error inside a step is caught, counted and reported, and the loop goes on with the next one. A step that fails does not kill the resident.

The stop flag is checked twice per turn. The second check is not redundant: a stop request wakes the loop precisely while it waits, and without a second look it would run one more step after being asked to stop.

Detached objects

A resident lives longer than any transaction, so it cannot keep a materialized object: such an object resolves its state in the transaction it was read in and is only good until that one ends. A SoilDetachedObject is the handle for this case. It holds the real object together with its object id, forwards what it does not understand, compares by object id, and writeBack writes a new version of the object in a transaction of its own. The subclass SoilDetachedResidentObject goes one step further: when it changes it tells its resident (noteChanged), which counts as work and wakes it up. A detached object without a resident stays silent.

The limits resident in the next section uses one, and its wiring follows a pattern. on:in: is called while there is a transaction, because only then the object and its id are known; it takes the real object, never the proxy that a cluster root is handed out as, since a proxy resolves through the transaction it was loaded in. resident: connects the detached object to the resident. The third step, instance:, gives the real object a back pointer to its detached object: the real object calls changed on itself, and the back pointer makes sure that the call goes to the detached object and from there to the resident, and not just to the object. The application only ever holds the detached object, and everything it sends to it reaches Limits unchanged.

There is a restriction to know about. The detached object is the root of one cluster, and the cluster can consist of more than one object, but writeBack writes that one cluster, as it is in memory, over the version in the database. Nothing is merged: what another transaction changed in between is overwritten. That is acceptable for state that only the resident changes, and it is why detaching is limited to one cluster for now. Detaching several clusters may become possible, but the benefit is doubtful: the state is still not synchronized back but overwritten, and every additional cluster is more that can be overwritten.

The limits resident

With steps, run strategies and detached objects in place, here is the limits resident, complete. The limits themselves are an ordinary object, the workspace of the resident. The application calls use on it:

Object << #Limits
	slots: { #used. #instance }

Limits >> used
	^ used ifNil: [ 0 ]

Limits >> use
	used := self used + 1.
	self changed

Limits >> changed
	instance changed

The description carries the length of the collecting window and creates the limits object when the resident is added:

SoilResidentDescription << #LimitsDescription
	slots: { #window }

LimitsDescription >> name
	^ #limits

LimitsDescription >> instanceClass
	^ LimitsResident

LimitsDescription >> window
	^ window ifNil: [ 30 seconds ]

LimitsDescription >> initializeFrom: aTransaction
	workspace := Limits new.
	aTransaction makeRoot: workspace

The resident detaches that object, wires it to itself, and writes it back when there is something to write:

SoilActiveResident << #LimitsResident
	slots: { #detachedLimits }

LimitsResident >> runStrategyClass
	^ SoilCoalescingRunStrategy

LimitsResident >> limits
	^ detachedLimits

LimitsResident >> prepareIn: aTransaction
	self runStrategy window: (self descriptionIn: aTransaction) window.
	detachedLimits := SoilDetachedResidentObject
		on: (self workspaceIn: aTransaction)
		in: aTransaction.
	detachedLimits resident: self.
	detachedLimits instance: detachedLimits

LimitsResident >> runStep
	detachedLimits writeBack

LimitsResident >> flushOnStop
	self basicHasPendingWork ifTrue: [ detachedLimits writeBack ]

initializeFrom: runs once, when the description is added, and creates the limits object. prepareIn: runs at every start: it sets the window from the description and detaches the limits object, as described in the previous section. What the application gets from limits behaves like the Limits object and is not tied to a transaction. use changes it in memory, without a transaction, and reports the change through changed, which reaches the resident as work.

The resident does not write on every change. The first change opens the window of the coalescing strategy, all changes in that window are collected, and one step at the end writes the object once. In a test with the window set to one second, ten processes called use 20000 times each, 200000 calls in all: the object in memory counted all of them immediately, the database still had 0, and after the window one step had written 200000. flushOnStop is called when the resident stops, and writes if a change is still pending, so a window that has not elapsed does not cost the counts collected in it. Only a crash loses something, the counts since the last write.

The log of that run, shortened:

resident limits: started
residents: ticking every 200 milliseconds, reporting every 10 seconds, for limits
resident limits: step 1
resident limits: 1 step since the last report, 0 errors in all
resident limits: stopping
resident limits: stopped

States and the two-phase start

A resident is always in one of six states.

States of a resident

failed and disabled are the two states a resident does not leave on its own; both answer isOutOfService. failed is what is left behind when a start throws an error, and the error stays on the resident so the embedder can read it. It is also where a resident ends up that has given up being revived, see below. disabled is the persisted flag on the description.

The start has two phases. The supervisor first loads, which creates every instance from the registry and starts none. Only then it starts them, each in its own protected block:

supervisor loadFrom: transaction.
supervisor
	startAllIn: transaction
	tickEvery: 1 minute
	reportEvery: 10 minutes

The split resolves dependencies between residents without declaring any. When the first one starts, all others exist already, so a resident that needs another one asks the supervisor for it by name (residentNamed:) at the moment it needs it and holds no reference. There is no start order and there cannot be a cycle. A resident that fails to start does not keep the others from starting; it ends up failed, visibly, and startAllIn: skips everything that is not created or is switched off in its description.

The start also starts the tick, a process of the supervisor that looks at every resident in the given interval. It revives a resident whose process has died and writes the periodic report. How it does that is a topic of its own; here it is enough to know that it exists.

Stopping without breaking anything

Stopping a resident by suspending or killing its process would leave a transaction half done and locks held. Soil stops cooperatively: requestStop only sets the flag and wakes the resident. The loop notices it between two steps, leaves, and marks itself stopped. A step that is running is finished first.

Stopping several residents is done in two phases as well:

Stopping in two phases

The supervisor first asks everybody to stop, then waits for the receipts. If it stopped them one after the other, a resident that is still running could talk to one that is gone already. After the timeout it passes in, a resident that has not answered is terminated. That is the deliberate hard way out and not the normal case: the image has to be able to quit, and a hanging resident must not prevent it. It is bearable because the database is transactional, a transaction that did not commit leaves no half state behind.

At the end every resident gets the chance to write what it collected (flushOnStop), each in a transaction of its own. For the limits resident this is the moment its last window is written.

Restart and disable

A resident that was stopped, that failed, or that was switched off comes back through one call, restartIn:. It clears the stop flag, reads the description again and starts. That is the same thing a first start does, there is no second mechanism. It is also why a change of configuration needs no watching: whoever changed the description calls restartIn: on the resident, because what has to happen after a change differs per field. For the limits resident a new window is a mere setter on the strategy, while a new way of counting would make the stored counts meaningless. That decision lies with the embedder that made the change.

Switching off works the same way. disabled is a flag on the description. To switch off, write the flag and restart, to switch on, clear it and restart. Since restartIn: reads the flag, no call can start something that the persisted flag says is off. How this looks is shown by a run of the test resident from Soil's own test package. Its process was killed twice, and it was set to give up after one revival (maxProcessRestarts):

resident active: process restarted (1)
resident active: gave up restarting active after 1 restart
resident active: stopping
resident active: stopped
resident active: started
...
resident active: disabled

The first death was revived by the supervisor's tick, the second exceeded the limit and left the resident failed. From then on the tick leaves it alone, until restartResidentNamed:in:timeout: brings it back.

What the embedder decides

Soil in version 6 does not hold the supervisor. The application that opens the database creates it (newResidentSupervisor), loads and starts it, stops it before closing, and passes in every duration that matters: the tick, the report interval, the stop timeout. It also does the ritual that adds a resident, because Soil provides the building blocks and not the sequence:

registry := self ensureResidentRegistryIn: txn.
registry addDescription: aDescription.
txn makeRoot: aDescription.
aDescription initializeFrom: txn.
txn commit

Whether a resident that is failed should keep the database from opening, and where the work of the steps is synchronized with the rest of the application (the supervisor offers a work gate as a meeting point and the embedder holds it), is likewise left to the embedder.

This is not where it is meant to stay. Integrating the supervisor properly into Soil is in progress, so that the application no longer creates, loads, starts and stops it, and no longer does the ritual of adding a resident by hand. It may already be part of the next Soil version.

The code is on github if you want to read ahead.

Found something worth flagging? Send feedback.
This work is licensed under CC BY 4.0.