Issue #012 — Build It Yourself #1, Part 1: A Crash-Safe Key-Value Store
Track: Build It Yourself | Vault: kv_store_part1.py + crash_test.py
Here is something that will change how you read every piece of database documentation: the most important primitive in database engineering is not a B-tree, not a WAL, and not MVCC. It is a crash-safe append. Three steps. One guarantee.
In 2012, engineers at a startup discovered their config store had been silently corrupting data for three months. The code wrote a JSON file on every config update. It worked perfectly across 4 million writes in staging. On the first power cut in production, the file was half-written. JSON parse failed. Service refused to start. Every engineer on the team had read about durability. None of them had built the primitive that makes it real.
By the end of this issue you will understand exactly what went wrong, why the fix is a single design pattern, and how that pattern underlies every durable storage system you have ever used — from SQLite to Postgres to RocksDB.
The problem with writing directly to a file
When your code calls write() on a file and the call returns success, the data is not on disk. It is in the Linux kernel’s page cache — a pool of RAM managed by the OS, separate from your process memory. The kernel decides when to flush it to actual storage. Usually this happens within seconds. But if the machine loses power before that flush, your data is gone. The OS confirmed the write succeeded. The disk never saw it.
This is not a bug. It is how write() is specified. Performance would collapse if every write() forced a physical disk write.
Databases need stronger guarantees. They use fsync() — a system call that blocks until the OS confirms the data has physically reached the storage device. Expensive: 500µs–5ms on NVMe, 10–30ms on spinning disk. But it is the only way to guarantee durability.
fsync() solves the write-to-cache problem. But it introduces a new problem: what if the machine crashes halfway through writing a record? You now have a partially-written file on disk. Corrupted. Unrecoverable.
