How to learn databases properly — and how this newsletter will take you there
Most engineers learn databases backwards — from the interface down. We’re going the other direction. Here’s the map, the philosophy, and exactly what you’ll be able to do by the end.
Bytes & B-trees | Pre-launch #2 of 3 | Free to read
There’s a particular kind of database problem that trips up even experienced engineers. Not the easy ones — slow queries, missing indexes, bad schema decisions. The harder kind: a transaction that committed but whose data was partially lost after a failover. A replication lag that spiked mysteriously during a schema migration. A Cassandra cluster that returned stale reads even though you configured quorum. A Postgres table that autovacuum couldn’t keep up with despite running constantly.
These problems share a pattern. They’re invisible to engineers who learned databases from the interface down — from SQL syntax, to ORM usage, to “add an index when slow.” They’re immediately understandable to engineers who know what’s happening inside the engine. The difference between the two isn’t years of experience. It’s a specific kind of knowledge: the internals.
That knowledge is what this newsletter builds, systematically, over three years.
The learning philosophy
Every concept in this newsletter follows the same sequence. We start with the problem — the specific constraint or failure mode that motivated someone to invent the mechanism in the first place. We ask: what would happen without it? What breaks? What becomes impossible?
Only after that question is fully answered do we introduce the solution. This order matters. The mechanism only makes sense in light of the problem it solves. Presented the other way — mechanism first, motivation second — it’s just memorisation.
Then we go into the implementation: how the mechanism actually works, with real code, real configurations, and real data structures. Not pseudocode. Not simplifications that obscure how it actually behaves. The real thing — or close enough that the difference doesn’t matter until you’re reading source code.
Then we anchor it in a real system — Postgres, RocksDB, Cassandra, DynamoDB, whichever shows the concept most clearly in production code.
Then we examine the trade-offs: what did this design gain, and what did it give up? What workloads does it help? What workloads does it hurt? When would you choose differently?
The goal is not that you can recite how MVCC works. The goal is that when your Postgres tables start bloating, you know immediately that it’s a vacuum problem, and you know exactly why — because you understand what MVCC is doing under the hood and what vacuum’s job is. That’s the difference between knowledge and understanding.
The three-year arc
Year 1 — Foundations (Issues #001–056)
What you’ll be able to do by the end:
By the end of Year 1, you understand why databases exist, how data is physically stored and retrieved, what ACID actually guarantees at the hardware level, how indexes are structured internally, how queries are parsed and executed, and how crash recovery works. You’ll have built a working key-value store and a minimal query engine from scratch. You’ll be able to look at any database’s core behaviour and understand the first-principles reason it works the way it does.
Topics covered:
Storage & I/O physics — bits, bytes, pages, disk geometry
B-trees & LSM trees — the two index structures that run the world
ACID from first principles — atomicity, durability, isolation at the byte level
MVCC & WAL internals — how Postgres actually manages concurrent transactions
SQL execution engine — parse, plan, execute, optimise
Database taxonomy — relational, document, key-value, columnar, graph, time-series
Build It Yourself #1: a key-value store (crash-safe writes → hash index → WAL)
Build It Yourself #2: a minimal SQL query engine (parser → logical plan → executor)
Year 2 — Internals & Scale (Issues #057–113)
What you’ll be able to do by the end:
You can dissect real database source code, design distributed architectures, choose between Raft and Paxos for a given use case, configure connection pools using queuing mathematics, and explain exactly why CockroachDB and Spanner make the trade-offs they do.
Topics covered:
Postgres buffer manager, query planner, and MVCC snapshot mechanism — at the code level
Distributed consensus — Raft, Paxos, FLP impossibility, the real implementations
Replication topologies — primary-replica, multi-master, leaderless, and when each breaks
Sharding strategies — range, hash, directory, hot keys, and resharding without downtime
NewSQL — CockroachDB, Spanner, TiDB — the ACID + horizontal scale problem
DuckDB & Apache Arrow — the unbundled analytics revolution
Cloud databases — Aurora, Neon, PlanetScale, Snowflake, ClickHouse
Queuing theory — Little’s Law, M/M/1 queues, and database capacity planning mathematics
Build It Yourself #3: a minimal replication log with primary and replica
Year 3 — Frontier & Ultra-scale (Issues #114–172)
What you’ll be able to do by the end:
You can design systems for billions of records, explain how vector search works at the index level, architect a production RAG pipeline correctly, read and critique database research papers independently, and synthesise three years of concepts into a coherent mental model you’ll use for the rest of your career.
Topics covered:
Vector search — HNSW, IVF, FAISS, approximate nearest neighbour
RAG as a database architecture problem — chunking, embeddings, hybrid search, re-ranking
AI-native query interfaces — NL-to-SQL, LLM query optimisation
Edge databases & local-first — SQLite, Turso, CRDTs, ElectricSQL
Streaming databases — Materialize, RisingWave, exactly-once semantics
Research frontiers — learned indexes, learned cardinality estimation, Bao
Company case studies — Netflix, Discord, Stripe, Uber, Cloudflare, Figma
Ultra-scale system design — 10B-record architectures, multi-region consistency
Event sourcing, CQRS, zero-downtime migrations, chaos engineering
The 3-year synthesis — every concept map and decision framework, compiled
How each week works
Every Sunday morning, a new issue arrives. The main piece — free for everyone — is 4,000–5,000 words covering the week’s concept from problem to mechanism to trade-off. The paid section includes: hands-on experiments you can run on your own machine, the Query Lab puzzle and its solution from the previous week, and (when applicable) the Build It Yourself implementation code.
The weekly production cycle
Day Activity Monday Research — primary sources, paper re-reads, code exploration, angle confirmed Tuesday Outline — section structure, diagrams listed, code examples scoped Wednesday–Thursday Write — full draft written, all code verified and run, diagrams built Friday Edit — technical accuracy check, clarity pass, Query Lab puzzle set Saturday Format — Substack layout, social thread drafted, GitHub repo pushed Sunday Ship — issue at 8am, thread at 9am, next issue outlined
The three quality tests every issue passes
Before any issue ships, it passes three internal tests:
1. The code test — every code sample must run without modification on a standard installation of the relevant system. No “you might need to adjust this.” It runs, or it doesn’t ship.
2. The newcomer test — could someone encounter this concept for the very first time in this issue and come away with a complete, correct mental model? Not a simplified one — a correct one.
3. The expert test — does an engineer who already knows this topic learn at least one non-obvious thing? If the answer to any of these three is no, the issue gets rewritten.
What the Query Lab is
Every issue ends with a Query Lab puzzle — an extreme SQL, Cypher, or reasoning challenge that forces you to apply the week’s concept directly. Not “write a SELECT statement.” More like: given this table size, page size, and row count, calculate exactly how many 8KB pages Postgres read before and after an index was added — and explain the I/O difference in terms of what the storage engine actually did.
Solutions — with multiple valid approaches and annotations on each — are published in the following issue’s paid section.
What to do before issue #001
Read the curriculum guide — Pre-launch #3, the next article in this series. It maps every one of the 172 issues with enough detail that you’ll know exactly what you’re signing up for before you subscribe. Then subscribe — free is fine to start.
If issue #001 isn’t what we described, unsubscribe immediately. We’d rather have readers who are genuinely here than subscribers who are being polite.
Issue #001 ships this Sunday.
“Data, storage & memory — first principles.” Free for all subscribers. Paid section includes four hands-on experiments and the first Query Lab puzzle.
Next: Pre-launch #3 — the complete curriculum guide. Every issue mapped, every track explained, every connection drawn. Long by design. Bookmark it.
© Bytes & B-trees · Substack · Weekly · bytesandbrees.substack.com