Course contents
On this page
Failure recovery and continuity — extended reference
Original detailed coverage and citations accompanying the focused failure recovery and continuity lesson.
This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.
Part 7: Building and operating applications on these engines
Prerequisites: Lesson 38 and the preceding foundations.
Learning outcomes
- Distinguish replica failover from backup-based recovery after logical corruption.
- Calculate an RPO and an end-to-end RTO including restore, replay, verification, and traffic return.
- Explain correlated failures, repair, and recovery evidence beyond the existence of backup files.
Replication faithfully copies a mistake
An operator deletes the wrong table. The deletion is valid and quickly reaches every replica. Automatic failover works perfectly but offers no older copy of the data. The service needs recoverable history outside the live replication path.
Failure recovery is a collection of capabilities rather than one backup switch. Hardware failover, corruption repair, accidental-deletion recovery, regional disaster recovery, and forensic reconstruction address different faults. The architecture must state which capability serves each scenario.
RPO and RTO
Recovery point objective, or RPO, specifies the tolerated loss of recent data under a stated failure. Recovery time objective, or RTO, specifies how long recovery may take. Both need a boundary: process restart, database acceptance, or restoration of the user-visible service at useful capacity.
An RPO of zero for acknowledged transactions under a machine loss requires an acknowledgment and replication strategy that actually survives that loss. A backup taken every hour does not independently establish zero loss. A short RTO requires more than having bytes available; the service must locate, restore, validate, and serve them.
Different incidents can have different objectives. A local process crash may recover in minutes, while a full regional loss requires a longer path. A logical corruption discovered weeks later needs retention long enough to recover the relevant point. Objectives should be tied to explicit failure scenarios.
Backup consistency
A physical backup copies engine representations such as pages and required metadata. A logical backup exports data through query or serialization interfaces. Each has different compatibility, restore speed, and fidelity properties.
Copying live files without a supported consistency protocol can produce a mixture that never existed as a valid database. A backup may need a start marker, consistent snapshot, archived log, or engine-specific coordination. The exact mechanism should follow the product’s documented backup interface.
A logical export can be portable but must preserve the intended snapshot, schema, constraints, sequences, and other required state. Restoring rows without ownership or privilege metadata can produce a different system. A complete recovery inventory includes extensions, keys, application versions, and routing configuration.
Point-in-time recovery
Point-in-time recovery combines a base backup with subsequent history, applying changes up to a chosen boundary. The engine must retain a complete required log sequence and interpret the stop point safely.
A timestamp near an accidental deletion may not uniquely identify the desired transaction boundary without additional evidence. Transaction identifiers, log positions, and audit records can help. Recovery should be validated in an isolated environment before replacing the live service.
PostgreSQL’s continuous-archiving documentation provides a concrete example of base backups plus WAL history. The general lesson is that a backup and its history are one recoverable set. A gap in the archived sequence can invalidate the desired recovery point even if most files are present.
Restore-time arithmetic
An illustrative 4 TB restore at an effective 500 MB/s takes 4000000/500 = 8000 seconds, about 2.22 hours, using decimal units. Replay, index work, integrity checks, key retrieval, provisioning, and cache warm-up add time.
Parallel restore can improve throughput only while storage, CPU, network, and format dependencies permit it. A raw device’s peak read rate is not the end-to-end restore rate. Small files, object-store request limits, decompression, and destination write amplification can dominate.
An RTO shorter than this estimate requires a different recovery architecture, such as prepositioned recoverable copies, faster restore paths, or a smaller required service subset. Setting a more ambitious timeout does not shorten the data movement.
Failover is not restoration
Failover promotes or routes to a surviving suitable replica. It can be fast because state is already present. It needs safe authority transfer and a history compatible with acknowledged outcomes. Restoration reconstructs state from backups or logs and usually has different latency and validation needs.
An asynchronous replica may meet a fast failover objective while losing recent writes under the relevant failure. A synchronous replica can preserve writes while adding normal-operation latency. Independent backups protect against logical mistakes that replicas propagate. These mechanisms complement rather than replace one another.
Split-brain prevention remains essential during recovery. A recovered old primary must not resume writing under obsolete authority. Fencing, membership, and routing need to be part of the procedure, not left to an assumption that the old machine stays off.
Corruption and repair
Checksums detect selected corruption; replicas and backups provide potential repair sources. A repair procedure must establish which copy is authoritative and avoid propagating corruption from the wrong source. Logical invariants can reveal errors that page checksums cannot detect.
A valid checksum on an incorrectly deleted record does not make the business state correct. Conversely, a corrupted page may contain mostly valid rows but still require engine-specific recovery. Operators should use supported repair tools and preserve evidence rather than editing bytes casually.
Repair can be resource intensive. Rebuilding indexes, comparing replicas, or restoring ranges competes with foreground traffic. A degraded cluster needs headroom and rate limits to repair without entering a second failure through overload.
Retention, access, and keys
Backup retention should cover the delay between a fault and its discovery. Short retention can make a perfectly executed backup program useless for an old unnoticed error. Long retention consumes cost and may preserve data the organization intends to delete under its policy.
Access to backups is access to data. Separate credentials, audit records, and protected encryption keys reduce the chance that one compromised role can erase or expose every recovery copy. An encrypted backup without recoverable keys is not a usable backup.
Format compatibility also matters. The restoring software must understand the backup, log, extensions, and schema. A disaster procedure depending on an unavailable old binary or undocumented secret can fail despite intact storage. Periodic restore verification tests the entire dependency chain.
A complete recovery sequence
A reviewable procedure identifies the incident boundary, freezes or fences unsafe writers, selects a recovery point, provisions isolated capacity, restores data and keys, replays required history, validates invariants, updates authority and routing, and observes service behavior after cutover.
Validation should include representative reads, ownership and accounting invariants, indexes, and known operation outcomes. It should not assume that process startup proves correctness. Reconnecting clients can create bursts, so admission and cache-warming policy belong in the recovery plan.
Recovery begins before the incident
A recovery procedure depends on state created during normal operation: valid backups, complete log archives, keys, configuration, compatible software, and enough independent capacity. The incident is too late to discover that one dependency was never retained.
A useful recovery inventory treats these as one set. Database bytes without the encryption key are not recoverable. Rows without required schema or extension versions may be uninterpretable. A base backup without its required log cannot reach the desired point. A restored database without safe routing and authority transfer can conflict with the old primary.
The service should define recovery classes. A process crash uses local recovery. A lost machine uses a suitable replica or restored copy. A logical deletion uses historical state. A compromised account can require credential revocation and forensic preservation before normal service resumes. One procedure cannot safely collapse all these incidents into “promote the backup.”
A backup manifest is a dependency graph
Suppose backup B consists of data objects, schema metadata, an initial log position, an ending position needed for consistency, and an encryption-key version. A manifest records those dependencies and their identities. Each object can be verified for length and checksum, while the manifest is published only after the required set is complete.
backup generation B
-> data objects with checksums
-> schema and engine format versions
-> required log interval [L_start, L_consistent]
-> encryption key version K
-> restore instructions and verification evidenceFigure: A recoverable backup is a complete dependency set, not merely a directory containing many plausible files.
A missing middle log segment is not repaired by having later segments. A checksum verifies object integrity against its expected content but does not prove the manifest describes a transactionally valid boundary. Use the engine’s supported backup protocol to establish consistency.
Retention deletion must follow dependencies. Removing an old base backup can be safe only if no retained recovery objective depends on it. Removing a key version can invalidate every backup encrypted with it. Storage lifecycle rules should be reviewed against the recovery graph rather than selected solely by object age.
Physical and logical recovery trade different work
A physical backup preserves engine-level representation and can restore large datasets efficiently when format and software compatibility permit it. A logical export re-expresses data through a higher-level interface, which can improve portability but require substantial parsing, constraint checking, and index construction during restoration.
Neither is inherently complete. A logical export may omit roles, extensions, sequences, or external configuration unless explicitly included. A physical backup may be tied to a particular engine format or topology. The intended recovery procedure determines which representation is useful.
For a hypothetical 2 TB logical export loaded at 100 MB/s, byte ingestion alone takes 20,000 seconds, about 5.56 hours. Index creation and validation can add more. A physical image transferred at 500 MB/s has a 4,000-second byte floor, about 1.11 hours, under those assumptions. These are workload models, not a universal physical-versus-logical speed ranking.
A team can retain both for different failure classes. A fast physical restore may serve ordinary disaster recovery, while a logical export supports migration or selective reconstruction. The operational burden is justified only if those recovery paths are tested and maintained.
Point-in-time recovery needs a stop decision
Imagine an accidental deletion at 14:03. The team has a base backup from midnight and a complete log archive. Restoring to “14:02:59” may be reasonable, but time alone can be ambiguous when concurrent transactions span that interval. Audit information and engine log positions help locate the intended boundary.
The team should restore into isolation first. It checks schema, row counts, selected business invariants, and the presence of operations known to precede the deletion. It also determines which valid later operations would be lost by replacing live state with the old snapshot.
A whole-database rewind can discard legitimate work after the chosen point. Selective repair may instead copy recovered records into the current system under a controlled transaction, but that requires reconciling current identifiers, references, and later changes. There is no universal automatic answer; the recovery contract and business evidence guide the procedure.
PostgreSQL’s continuous-archiving documentation gives a concrete base-backup-plus-WAL model. The general lesson is the completeness and ordering of the history, not a claim that every engine shares its file procedures. Continuous archiving and PITR
RTO is a sum of operational stages
A recovery-time objective should include detection, diagnosis, approval or automation delay, provisioning, byte transfer, replay, validation, routing, and warm-up. Reporting only transfer speed understates user-visible recovery.
Assume a hypothetical incident spends 5 minutes detecting and diagnosing, 10 minutes provisioning, 40 minutes transferring, 15 minutes replaying, 10 minutes validating, and 10 minutes warming and routing. Total is 90 minutes. Halving transfer time saves 20 minutes, making total 70 minutes, not 45. Other stages remain.
Some work can overlap. Provisioning compute and locating keys may happen concurrently. Validation of completed segments can overlap transfer when safe. A dependency diagram identifies real opportunities rather than simply summing or arbitrarily overlapping everything.
The recovery objective should name the finish line. Database accepts a connection, essential transactions meet a reduced target, and all clients regain normal performance are different points. A service can elect a leader quickly yet take many minutes to warm caches and reconnect subscribers.
RPO is tied to acknowledgment and retained history
A backup frequency alone does not establish the loss window if continuous logs are also archived. Conversely, frequent backups do not guarantee a small RPO if their completion is unverified or archives have gaps.
Suppose base backups occur daily and logs are independently durable every minute under a stated policy. A regional loss may preserve history much newer than the base backup, subject to actual archive completion and dependencies. The RPO should be measured from surviving recoverable history, not the schedule printed in configuration.
For acknowledged writes, zero loss under one failure requires the normal commit contract to preserve enough state outside that failure domain. It does not promise survival of every conceivable correlated event. Naming the failure model avoids treating “zero RPO” as an unlimited physical guarantee.
Applications can also contribute durable evidence. An external payment provider may retain a charge that the database lost under a weak acknowledgment policy. Reconciliation can identify the mismatch, but it does not retroactively make the original data contract stronger. Recovery must account for such external facts before accepting new work.
Replica repair requires an authoritative source
When copies disagree, repair needs a rule establishing which history is valid. Choosing the newest wall-clock timestamp is not generally sufficient. A stale leader can write later timestamps, clocks can differ, and logical corruption can be recent.
Consensus histories, version vectors, checksums, and application-level evidence solve different parts of the problem. A page checksum detects bytes that differ from their expected integrity value. It does not prove that the business record was authorized. A replicated valid deletion can be byte-perfect and still require historical recovery.
Before repair, preserve evidence needed to understand the fault. Overwriting the only differing copy can destroy forensic information. A supported repair tool can reduce risk, but the operator must know whether it repairs physical corruption, replica divergence, or logical state.
For an LSM-style replicated system, deletion retention and repair schedules interact. Old values on a long-absent replica can return if deletion evidence is removed before convergence is safe under the protocol. Operational retention is part of correctness.
Restore validation must test meaning
Byte checksums and successful engine startup are necessary evidence but insufficient for application recovery. The restored system should satisfy selected invariants: unique ownership, balanced ledger totals under the accounting model, valid references, monotonic sequence expectations, and known operation outcomes.
An invariant check can be expensive. It may need to run before cutover, on a representative subset with a justified sampling policy, or as a staged verification while service remains restricted. The choice depends on the consequence of serving incorrect state. A financial ledger and a disposable cache deserve different acceptance criteria.
Validation also tests authorization. Restored roles or policies can differ from current application expectations. A backup from before a user revocation may contain old permission state. Recovery must determine which security facts should be restored historically and which must remain current. Blindly replaying old credentials can reopen access.
The restored application version matters. New code may expect fields absent from the old recovery point. A recovery runbook should specify compatible code or migration steps, while avoiding a migration that overwrites evidence before validation.
Independent failure domains include people and credentials
A backup on the same host protects against selected file mistakes but not host loss. A replica under the same compromised administrative credential may be deleted with the primary. A remote archive with independent retention controls can protect against a different class of incident.
Independence is never absolute. Regions can share identity systems, key services, software releases, or organization-wide operator actions. A recovery design should identify likely correlated causes and avoid relying on copies that fail together under the incident it claims to cover.
Access to backups is itself sensitive. Backups often contain all tenants and older deleted data. Encryption, narrowly scoped restore permissions, audit logs, and controlled export paths reduce exposure. Recovery should not require granting every responder unrestricted permanent access.
A disaster can also remove the people who normally operate the service. Documentation, rehearsals, and alternate access paths improve organizational recoverability. The procedure must be usable by the designated responders, not only by the original author who remembers undocumented details.
A regional-loss scenario
A hypothetical region A contains the primary authority, local replicas, and the latest warm cache. Region B retains a recoverable replica and independent archived logs. A fails completely. The procedure first establishes the surviving committed history and fences old authority, then makes B authoritative under the cluster protocol.
Client traffic shifts to B with randomized reconnect and bounded admission. B warms state while applying any required history. Reports and optional exports pause so essential transactions retain capacity. The team measures the degraded latency target and begins rebuilding redundancy in a permitted location.
When A returns, it joins as a recovering participant rather than assuming its old role. Its state is compared under the authoritative history and rebuilt where necessary. Direct clients using stale routes must be rejected or redirected. The old region’s survival does not invalidate the authority transition.
This is failover, not historical restoration. If the original incident was a replicated logical deletion, the same steps would preserve the deletion. The team must diagnose the fault class before selecting the recovery tool.
A recoverability acceptance statement
A useful statement is concrete: for machine loss, acknowledged operations survive and essential service returns within a tested interval; for logical deletion discovered within a retained window, an isolated restore can recover the chosen point; for region loss, service resumes under a specified degraded capacity and locality contract.
Each clause needs evidence from a rehearsal or an explicitly untested assumption. A backup success metric reports creation, while a restore rehearsal reports usability. Both belong in operations. Recovery is an architectural capability only when its dependencies and user-visible outcome are known.
A recovery timeline separates loss from downtime
last surviving recoverable position P ---- incident I
<---- RPO window ---->
incident I -> detect -> provision -> restore -> replay -> validate -> route
<------------- RTO interval --------------------->Figure: RPO concerns the gap in recoverable history; RTO concerns the time to regain the defined service. The two can improve independently.
A replica can shorten restore time because bytes are already present, yet lose recent acknowledged data under an asynchronous policy. An independent archive can preserve a newer point while requiring a longer restoration. Neither metric alone describes recovery quality.
The service return marker must also be explicit. If routing completes but most subscriptions still need snapshots, the database may be available before the product is useful. A reduced essential-service objective can be legitimate, provided it is named and measured separately from full restoration.
A rehearsal should retain the actual timings and evidence for every stage. It can reveal that key retrieval or validation takes longer than transfer, or that the presumed parallel steps share one constrained network. Those findings change the architecture more usefully than a checklist showing that backup creation succeeded.
Recoverability is a maintained capability. Changes to schema, encryption, storage layout, deployment, and data volume can invalidate an old rehearsal. The procedure and evidence should evolve with the system whose survival they are meant to establish.
A recovery objective becomes credible when the exact retained dependency set has produced a usable service under a rehearsed failure, with remaining assumptions recorded explicitly.
Design conclusion
Recovery engineering turns objectives into tested paths through data, metadata, keys, software, and authority. The useful question is not whether backups exist, but whether the system can restore the required state within the promised loss and time bounds.
Lesson 40 examines security as another cross-cutting property of these paths. Backup, restore, replication, caches, and diagnostics all participate in the same data-protection boundary as ordinary queries.