Course contents
On this page
Lesson plan 11: Log-structured storage
Trace writes through a mutable table, immutable sorted files, and compaction.
Placement: Part 2: Representing and locating data.
Prerequisites: Lesson 10: Tree and hash indexes, plus the earlier dependency chain.
Learning outcomes
- Trace writes through a mutable table, immutable sorted files, and compaction.
- Explain Bloom-filter false positives, tombstone retention, and version selection on reads.
- Calculate read, write, and space amplification and connect them to background capacity debt.
- Separate a general mechanism, a documented implementation choice, and an illustrative numerical assumption.
Teaching sequence
- Establish the motivating situation: A write-heavy service runs quickly until compaction debt consumes its disk bandwidth.
- Introduce the vocabulary and the correctness obligation before an optimization. Define memtables, SSTables, Bloom filters.
- Develop the mechanism: Follow a mutation through log, memtable, flush, manifest, and compaction; derive when tombstones can be removed safely.
- Follow the complete worked example: 30 MB/s of logical writes at total write amplification 8 consumes 240 MB/s of device writes. Explain the assumptions, intermediate units, and limiting interpretation.
- Extend the scenario to changed load, skew, or failure, explaining compaction, tombstones, amplification.
- Connect the mechanism to a documented implementation and an alternative with a different workload fit. Close by stating the remaining cost and the next lesson dependency.
Required chapter material
Use a complete execution trace or worked calculation, supported by an annotated static diagram or table. Include language-neutral pseudocode or labeled SQL when it clarifies the mechanism. State preconditions, ownership, visibility, and failure boundaries for algorithms. Describe operational costs as well as the normal path. All questions raised in the narrative must be resolved in the text.
Visual explanation
A read/write path shows WAL, memtable, sorted runs, and compaction; a version trace demonstrates when removing a tombstone would resurrect deleted data. The accompanying prose explains the arrows, state changes, or table entries so the lesson remains understandable in a plain Markdown reader.
Editorial checks
Confirm that the numerical example agrees with its units and assumptions. Check that a faster path preserves the stated contract, or explicitly identify the changed guarantee. Keep hypothetical figures separate from product measurements. Explain both an appropriate use and a limitation of the case design. Check terminology against the glossary and links against the course index.