Course contents
On this page
Lesson plan 39: Failure recovery and continuity
Distinguish replica failover from backup-based recovery after logical corruption.
Placement: Part 7: Building and operating applications on these engines.
Prerequisites: Lesson 38: Geographic architecture, plus the earlier dependency chain.
Learning outcomes
- Distinguish replica failover from backup-based recovery after logical corruption.
- Calculate an RPO and an end-to-end RTO including restore, replay, verification, and traffic return.
- Explain correlated failures, repair, and recovery evidence beyond the existence of backup files.
- Separate a general mechanism, a documented implementation choice, and an illustrative numerical assumption.
Teaching sequence
- Establish the motivating situation: A replicated deletion reaches every live replica; only an independent recoverable history can restore the data.
- Introduce the vocabulary and the correctness obligation before an optimization. Define backup, restore, PITR.
- Develop the mechanism: Distinguish hardware failover from logical recovery and calculate the complete recovery path.
- Follow the complete worked example: Restoring 4 TB at 500 MB/s takes 8000 s, about 2.22 h, before replay and verification. Explain the assumptions, intermediate units, and limiting interpretation.
- Extend the scenario to changed load, skew, or failure, explaining repair, failover, disaster recovery, RPO, RTO, correlated failures.
- Connect the mechanism to a documented implementation and an alternative with a different workload fit. Close by stating the remaining cost and the next lesson dependency.
Required chapter material
Use a complete execution trace or worked calculation, supported by an annotated static diagram or table. Include language-neutral pseudocode or labeled SQL when it clarifies the mechanism. State preconditions, ownership, visibility, and failure boundaries for algorithms. Describe operational costs as well as the normal path. All questions raised in the narrative must be resolved in the text.
Visual explanation
A recovery timeline marks the last recoverable position, incident, restore, replay, validation, and service return, with separate RPO and RTO intervals. The accompanying prose explains the arrows, state changes, or table entries so the lesson remains understandable in a plain Markdown reader.
Editorial checks
Confirm that the numerical example agrees with its units and assumptions. Check that a faster path preserves the stated contract, or explicitly identify the changed guarantee. Keep hypothetical figures separate from product measurements. Explain both an appropriate use and a limitation of the case design. Check terminology against the glossary and links against the course index.