Issue-014: Build It Yourself #1 — Part 3: Checkpointing and fast startup
Parts 1 and 2 fixed correctness and read performance. One problem remains: every restart scans the entire log to rebuild the index. For a 300,000-record log, that’s 600ms before the database can serve a query. Postgres, Redis, and Bitcask all solve this the same way — periodic checkpoints. Today we implement it in 80 lines.
There is a pattern in storage systems so fundamental that once you see it, you see it everywhere:
Keep an append-only log. Never modify history.
Periodically snapshot the current state. Call this a checkpoint.
On startup, load the snapshot, then replay only what came after it.
Postgres does this with checkpoints and WAL replay. Redis does this with RDB snapshots and AOF replay. Bitcask does this with its “keydir” snapshot and log replay. The specific names differ. The pattern is identical. This issue implements it in about 80 new lines on top of Part 2.
After a checkpoint on a 300,000-record log, startup drops from 613ms to 202ms — a 3× speedup. More importantly, the startup time no longer scales with total log history. It scales only with records written since the last checkpoint. Checkpoint every 10,000 writes and startup is bounded regardless of how long the system has been running.
What a checkpoint stores
Snapshot file format:
[MAGIC: 4 bytes] "BBKV" — identifies this as our format
[checkpoint_offset: 8 bytes] byte position in log when snapshot was taken
[entry_count: 4 bytes] number of index entries
for each entry:
[key_len: 4 bytes]
[key bytes]
[value_offset: 8 bytes] same as what's in self._index
[value_len: 4 bytes]
It’s a complete serialisation of self._index plus the log byte offset at the moment checkpoint() was called. That offset is the crucial piece — it tells us exactly where to seek in the log on startup to replay only the new records.
