Soil: identity and the heap
The last post ended with two loose ends. A reference from one cluster to another is stored as an "id", but I never said what an id is. And I kept talking about clusters "on disk" without saying where on disk, or how you get from an id to the actual bytes. Both ends meet in the same place, so let's tie them off together.
Why a reference can't just be a position
The tempting answer is that a reference is a position: the byte offset in the file where the target cluster sits. Store the offset, seek to it, read. It even seems to fit, because writing a cluster naturally gives you back the position where it landed.
It falls apart the moment you change something. Soil's object file is *append-only*: it never overwrites. When you modify an object and save it, its new bytes are appended at the end of the file, at a new position, and the old bytes stay where they were. That is a deliberate choice, and a later post on versioning will show why it's a good one. But it means an object's position is not stable. If references stored positions, every reference to an object would break the first time that object was rewritten somewhere else.
This isn't a problem unique to disk, either. A virtual machine with a moving garbage collector runs into the same wall: a compacting collector slides live objects together to reclaim space, so an object's address in memory changes over its lifetime. A raw pointer to that address would go stale the moment a collection ran, which is exactly why such runtimes don't let references be bare memory addresses - they add a layer of indirection instead, so the reference stays valid while the collector is free to move the bytes underneath it. Soil's problem is the same shape, just on disk instead of in memory.
So a reference needs to name *which object* it points to in a way that never changes, even as the object's bytes move around the file. It needs a stable identity. That is what an object id is.
What an object id is
A `SoilObjectId` is eight bytes, and it splits into two parts:
SoilObjectId>>asByteArray
^ (segment asByteArrayOfSize: 2), (index asByteArrayOfSize: 6)The first two bytes are a *segment* number, and the remaining six are an *index* within that segment. Think of the segment as which drawer the object belongs to, and the index as its numbered slot in that drawer. Nothing in those eight bytes says anything about where the object physically sits in the file, and that's the whole point: the id is a name, not an address.
*A SoilObjectId is a name, not a location: two bytes of segment, six bytes of index.*
The 2/6 split itself is just a fixed choice today, not a knob you can turn. A later version of Soil could plausibly make it configurable per database - trading segment range for index range depending on whether you expect many small segments or a few huge ones - as something you'd set once when the database is created. For now it's baked in.
A couple of details fall out of this. The root object, the one known entry point into the database, has the fixed id segment 1, index 1, so everyone can find it without looking anything up. An index of 0 means "not allocated yet", which matters because ids are handed out before the real slot is assigned.
That zero-index case also decides how two `SoilObjectId`s get compared. Once an id has a real index, comparing two of them is a value comparison: same segment, same index, same id, regardless of how many separate `SoilObjectId` objects in memory happen to represent it. A placeholder id is different. It's minted fresh inside a transaction before any slot exists for it, so it carries index 0, and every other placeholder created around the same time looks exactly the same from the outside: same segment, same index 0. Comparing those by value would call two unrelated placeholders equal just because they haven't been assigned yet. So whenever the index is 0, comparison falls back to identity (`anId == otherId`) instead: a placeholder is only equal to itself, never to another placeholder that merely looks the same.
Segment 0 is special: it's the *meta* segment, where the database keeps information about classes and schema rather than your objects. We'll need the meta segment in a later post; for now just know segment 1 upward is where your data lives.
The heap: append and never look back
The objects themselves live in what Soil calls the object heap, a file whose header starts with the marker `SOIL|OBJECT`. Writing a cluster to it is the simplest operation in the whole system: append the bytes to the end, and get back the position where they started.
There is no update-in-place, ever. Saving a new version of an object appends a fresh copy and leaves the previous bytes untouched. An append-only file like this is easy to reason about, cheap to write to, and safe under a crash in ways we'll lean on later. The obvious question, "doesn't the file just grow forever with old versions?", is a real one, and it has an answer, but not in this post.
The catch is the one we already saw: if positions keep changing, how does anyone find the current version of an object from its stable id?
The index: from id to position
The answer is a second file, the object index, whose header reads `SOIL|OBJECT INDEX`. Its only job is to map an object id to the position of that object's current bytes in the heap.
It's laid out as a flat array of fixed-size slots, one per object, so the lookup is direct arithmetic rather than a search: an object's index number *is* its slot number, and the slot holds a heap position. Each slot is 8 bytes, so an id's index times 8 gives its byte position in the index file directly, no search needed - follow segment and index to the right slot, read the position, seek there in the heap, read the cluster.
*The id stays fixed; the position behind it moves. Updating an object appends new bytes and rewrites just the one slot to point at them.*
Updating an object is now two writes: append the new bytes to the heap, then overwrite that object's slot in the index with the new position. The id never changes, the slot is rewritten in place, and the position it points to walks forward through the file. The old bytes are still sitting there, orphaned by the index but not gone.
This is exactly what the proxy from the last post does when you finally touch it. The proxy is holding an object id. Resolving it means going to that id's slot in the index, reading the position, and reading the cluster from the heap at that position. Identity, index, heap: a proxy is just those three steps waiting to happen.
Where this leaves us
An object id is a stable name, split into a segment and an index. The heap is an append-only file of cluster bytes. The object index maps each id to the current position of its bytes, and an update appends to the heap and repoints one slot. From an id you can now always find the object, no matter how many times it has been rewritten.
Two things I've deliberately deferred are now visible on the horizon. The heap keeps every old version instead of overwriting, and the index quietly points past them. That is not waste, it's the foundation of how Soil gives each transaction a consistent snapshot without locking. Getting there means talking about transactions first, which is the next post.
One more thing worth being honest about: segments are still mostly a placeholder. Every object you store today lands in segment 1, segment 0 holds meta information, and that's it, two segments in the whole scheme. That's not because two is the natural number of segments, it's because we didn't want to commit to more before there was a real reason to. Splitting objects across segments opens the door to things like per-segment ownership and permissions, or to segments that simply work differently from each other - one day, perhaps living in shared memory, or served from somewhere else on the network instead of the local heap file. None of that exists yet. But putting a segment number in the id from day one keeps that door open; adding it later, once ids without any notion of segment were already out in the world, would have been much harder.
The code is on github if you want to read ahead.