Observability and verification — extended reference

This reference preserves the earlier detailed chapter and its citations. Start with the guided explanation, then use these sections for additional mechanisms and qualifications. References to the original lesson sequence are historical.

Part 7: Building and operating applications on these engines

Prerequisites: Lesson 41 and the preceding foundations.

Learning outcomes

• Connect tracing, resource profiles, latency distributions, and correctness histories to a stated SLO.
• Explain workload, crash, fault-injection, and recovery tests and what each can establish.
• Distinguish a passing example from proof of correctness across all possible executions.

A latency graph is an observation, not an explanation

A dashboard shows p99 commit latency increasing. CPU appears normal, and query execution spans are short. The missing evidence is a growing storage queue delaying durable-log acknowledgments. Another incident with the same graph could be caused by lock waits or network retransmission. Effective observability connects the symptom to the resource and correctness boundary responsible.

Verification asks whether the system preserves its promised outcomes under normal execution and faults. Monitoring asks what is happening now. Profiling asks where resources are spent. Tracing follows dependencies. These tools overlap, but each answers a different question.

Define user-facing objectives

A service-level indicator measures a property such as successful response rate or latency at a stated boundary. A service-level objective sets a target over a window. The measurement must include relevant errors and timeouts rather than reporting only successful fast responses.

For an illustrative 99.9% time-based availability objective over a thirty-day month, the unavailable allowance is 30 × 24 × 60 × 0.001 = 43.2 minutes. A request-based objective uses a different denominator and cannot be interpreted as the same time allowance without assumptions.

Objectives should distinguish operation classes and failure modes. A catalog read and a durable transfer may have different latency targets. A service can be available but return stale data outside its contract, so correctness and freshness indicators may also be needed.

Metrics and saturation

Useful resource metrics include CPU demand, runnable queues, memory ownership, cache misses, device queueing, log flush time, replication lag, lock waits, compaction debt, and output backlog. Aggregate utilization can hide a single saturated shard or core.

Counters should separate offered attempts, admitted work, commits, aborts, rejects, retries, and delivered updates. A falling request rate during a slowdown may indicate client backoff or a closed-loop bottleneck, not recovery. A growing backlog needs an age distribution as well as a count.

High-cardinality labels can make monitoring expensive. Tenant or key detail is useful for hotspots but should be sampled or aggregated appropriately. The observability system must not become a new unbounded workload or leak sensitive identifiers unnecessarily.

Tracing the critical path

A distributed trace should include connection-pool wait, network calls, admission, lock or intent waits, execution, durable acknowledgment, and downstream dependencies where relevant. A span’s start and end must have a clear meaning.

Parallel child spans overlap; adding their durations overstates elapsed time. Missing asynchronous links can hide the actual dependency. Clock skew can distort cross-machine timing, so causal relationships and measured durations should be interpreted carefully.

Trace data can contain SQL parameters, payloads, and identities. Redaction and access controls are part of the design. A useful trace records enough structure to locate the bottleneck without indiscriminately storing secrets.

Profiling and controlled experiments

CPU profiles reveal execution hotspots; allocation profiles reveal memory churn; lock profiles reveal waiting; storage and network traces reveal other paths. A low CPU percentage does not mean a request is cheap if it waits for a serialized resource.

A controlled experiment changes one relevant factor while preserving workload and guarantees. Turning off durable flushing may explain the cost of persistence, but the resulting throughput is not the same service contract. Moving clients closer changes topology and should be reported.

Observational evidence supports hypotheses. If compaction and p99 rise together, the mechanism might be device contention, cache displacement, or another shared cause. Targeted measurements distinguish them. A configuration change that coincides with recovery is not automatically the cause.

Correctness histories

A history records operation invocation, completion, result, and identity. A checker compares it with a model such as a linearizable register, transactional account invariant, or unique-item ownership. Incomplete operations remain uncertain and must be interpreted under the model.

A test should not declare a timed-out write absent merely because no success response was recorded. It may have committed. Conversely, a success acknowledgment should be checked against the guarantee it claims, including after relevant failures. Operation identities make the comparison more precise.

Jepsen’s consistency-model material supplies useful vocabulary, while formal papers define the underlying contracts. A finite test cannot prove all executions correct, but it can reveal violations that throughput tests never examine.

Crash and fault injection

Fault tests interrupt execution at meaningful boundaries: before log durability, after commit before reply, during page write, during leader change, during snapshot installation, and during membership transition. Network delay, partition, duplication through retries, process pause, disk-full, and failed writes explore different assumptions.

The test oracle checks invariants and accepted outcomes after recovery. Process startup alone is not enough. Index consistency, ownership, accounting, and retained operation results can reveal silent errors. Repeated crashes test whether recovery progress is itself safe.

Deterministic simulation can explore many schedules reproducibly by controlling time, network, and faults. FoundationDB’s architecture paper describes deterministic simulation as an important part of its engineering approach. Reproducibility helps turn a rare failure into a debuggable execution rather than an anecdote.

Workload verification

Load tests should include warm and cold states, steady-state maintenance, skew, long transactions, slow consumers, and degraded replicas. An open-loop arrival model exposes backlog under sustained demand; a closed-loop concurrency sweep answers another useful question. Both need generator-capacity checks.

Record the complete environment and contract. Hardware, versions, topology, isolation, durability, replication, dataset age, and outstanding-work limits can all change the result. A benchmark should be reproducible enough that another engineer can identify what task was actually measured.

Derived numbers need unit checks. MB and MiB differ; throughput and in-flight count are not interchangeable; percentiles do not average or add like means. Small arithmetic errors can produce large architectural mistakes when multiplied across tenants or replicas.

Incident interpretation

A useful incident record separates observations, hypotheses, actions, and confirmed mechanisms. It states the user impact and the correctness boundary at risk. Recovery should preserve evidence where feasible without delaying urgent safe action unnecessarily.

Post-incident changes should address the mechanism: admission for overload, fencing for stale authority, retention for recoverability, or resource isolation for noisy neighbors. Merely increasing a timeout can reduce visible errors while worsening waiting and resource retention.

Define the measured operation before collecting metrics

A database request can be timed from client intent, connection acquisition, server arrival, execution start, durable commit, or response receipt. Each boundary answers a different question. A system reporting 1 ms execution and 100 ms client delay may be functioning exactly as its narrow metric says while delivering a poor product experience.

For a hypothetical purchase, the service-level indicator can measure the fraction of valid purchase attempts that reach a definitive outcome within 200 ms from the client-facing entrypoint. Invalid credentials and sold-out responses may be classified separately according to the product objective. The definition should not exclude timeouts merely because they lack successful duration samples.

A separate correctness indicator records whether accepted purchases preserve ownership and deduplication rules. Availability and latency alone do not prove this. A system that quickly returns an incorrect success response can satisfy a superficial latency dashboard while violating its actual purpose.

The monitoring design should state its population, start, finish, exclusions, and aggregation window. That makes the objective reproducible and prevents convenient changes to measurement from masquerading as improvements.

Histograms preserve more than averages

An average hides distributions. Ten requests at 1 ms and one request at 1,000 ms average about 91.8 ms, but most users had a very different experience from the unlucky one. Percentiles summarize parts of the distribution, while a histogram retains enough shape to diagnose changes.

Percentiles generally cannot be averaged across servers to obtain the fleet percentile. A small slow shard and a large fast shard contribute different request counts. Aggregate compatible histograms or raw observations using a method appropriate to the monitoring system.

Sampling also matters. If traces sample only successful requests or discard long spans, the dataset excludes the incidents most worth studying. Tail-based sampling can retain slow or failed traces, while controlling the resource cost. Its selection rules should remain visible when interpreting results.

Clock behavior affects durations. Local monotonic clocks are appropriate for elapsed time, while cross-process spans need care with wall-clock alignment. A negative apparent network interval can be clock error rather than impossible causality. Causal identifiers and local durations help reconstruct the path without assuming perfect synchronization.

Trace one durable write to the constrained resource

A useful request trace separates pool wait, routing, lock or validation wait, execution, log append, durability wait, response construction, and delivery where instrumentation permits. It links those spans with resource metrics such as device queue depth and replication lag.

client request
  -> pool wait
  -> server queue
  -> protected execution
  -> log durability wait
  -> reply
       + correlated device latency / replica lag

Figure: A trace locates elapsed time; correlated resource metrics help explain why that stage slowed. A long span alone is a symptom.

Suppose commit latency rises while execution remains stable. Device synchronization latency and queue depth rise at the same time as a backup starts. That evidence supports a storage-contention hypothesis. It does not yet prove causation; a controlled change or additional profiling can distinguish competing explanations.

If execution time rises instead, query plans, row counts, cache misses, and CPU profiles become more relevant. One dashboard labeled “database slow” cannot choose between these mechanisms. Instrumentation should mirror the architecture’s dependency graph.

Metrics should expose occupancy and progress

Utilization reports busy time, saturation reports demand exceeding immediate service, and progress reports useful completion. A device can be fully utilized while meeting its target, or moderately utilized while latency suffers from rare stalls. Queue depth and service-time distributions add context.

For replication, track both bytes and history positions. A follower can be seconds behind with few bytes during quiet traffic, or many megabytes behind during a burst. Applied position and durable position can differ. The metric name should reflect which boundary it represents.

For subscriptions, socket count is not enough. Track initial-snapshot completion, queued bytes, client-applied progress where available, disconnects due to backpressure, and resynchronization. For MVCC, track old snapshot age and retained version bytes. These metrics identify resource owners rather than only total memory.

Background maintenance needs progress indicators too. Compaction debt, checkpoint age, archive lag, and free-space reserves can deteriorate long before foreground latency fails. Alerting on these trends can preserve the capacity needed for recovery.

Profiling distinguishes computation from waiting

A CPU profile shows where executing threads spend CPU time. A wall-time or off-CPU profile can show blocked or descheduled time under the profiler’s model. Confusing them can lead to optimizing code that is barely running while the request waits on storage.

Suppose a handler’s wall time is 100 ms, but it uses 2 ms CPU and waits 98 ms for a lock. Rewriting its serialization loop to save 0.5 ms cannot solve the dominant delay. The correct investigation follows the lock owner and why its protected interval is long.

Conversely, a high CPU fraction in allocation and copying can reveal avoidable representation costs. Reducing copies or batching may help more than adding threads. Memory bandwidth and cache misses can limit scaling before cores appear fully busy in a simplistic process metric.

Profiling itself consumes resources and may sample imperfectly. Production use should choose safe overhead and avoid exposing sensitive payloads. An observation is evidence with a collection method, not an infallible direct view of every event.

Load generation can hide overload

A closed-loop client waits for each response before issuing its next request. As the server slows, the client naturally sends fewer requests. That can make the benchmark look stable precisely because offered load falls during bad periods.

An open-loop generator follows an intended arrival schedule independently of response completion, within its own capacity. It can reveal queue growth and deadline failures. If the generator falls behind, it must report missed schedules or intended-start latency rather than silently shifting all arrivals later.

Assume intended traffic is 10,000 requests/s for ten seconds. The test should account for 100,000 scheduled arrivals. If only 60,000 were actually sent because the generator blocked, the result is not a successful 10,000/s test. Generator CPU, connection limits, and network can become the bottleneck.

Coordinated omission occurs when the measurement process underrepresents delays because it stops observing arrivals during a stall. The remedy is a clear arrival model and latency measured against intended demand where appropriate. No one measurement method suits every workload, but the chosen method must match the claim.

Benchmark the envelope, including skew

A useful workload matrix varies data size, cache warmth, read/write mix, transaction length, result size, contention, and subscriber fan-out. Uniform random keys are a useful baseline but cannot establish hot-key behavior.

Suppose one million keys are evenly accessed in a baseline. The engine scales across partitions. In production, 40% of writes target one key. The total dataset is unchanged, but the authoritative path is now concentrated. A benchmark that preserves only average key count misses the important workload property.

Durability and consistency must remain equal across comparisons. One configuration acknowledging from memory and another waiting for replicated durability are testing different contracts. One local historical read and one fresh distributed read are also different operations.

Report failed, rejected, retried, and timed-out attempts, not just successful throughput. Useful commits per second and attempts per useful commit reveal retry amplification. A system doing more internal work can look busier while serving fewer users.

Crash testing targets persistence boundaries

A crash test interrupts execution at meaningful points: before log append, during a record write, after append but before flush, after durability but before page installation, and after commit but before reply. Recovery should produce only outcomes allowed by the acknowledgment contract.

The test must distinguish process termination from machine power loss. Killing a process may leave operating-system buffers intact, while a power failure can lose them. A simulation or fault-injection framework must model the relevant persistence behavior rather than assuming every termination tests the same fault.

Torn writes, reordered persistence, disk-full errors, and corrupt checksums require specific handling. An engine that correctly recovers complete records can still fail when an incomplete page appears valid because metadata publication was ordered incorrectly. Fault points should follow the recovery proof.

Repeating the same happy-path restart many times is less informative than systematically exploring the boundaries where the algorithm changes durable state. The mechanism guides the test plan.

Distributed faults need histories, not only uptime

A partition test can show that the service stayed up while allowing conflicting authorities. Correctness verification records operations, invocation and completion intervals, results, and enough identity to compare the observed history with a model.

For a single register, a checker can ask whether completed reads and writes admit a linearizable ordering. For multi-record transactions, the model must represent the promised transactional semantics. A checker for one object does not automatically establish correctness of arbitrary cross-object invariants.

Jepsen’s consistency-model reference is useful for distinguishing these contracts. It should be treated as vocabulary and testing context rather than a claim that one successful test proves all executions correct. Consistency models

An operation with no response can have an uncertain outcome. A history checker must model that uncertainty rather than assume every timed-out write aborted. Stable operation identifiers make both application recovery and test interpretation more precise.

Deterministic simulation changes debugging economics

Distributed bugs often require a rare combination of message delay, process pause, disk behavior, and leadership transition. A deterministic simulator can control the schedule and replay a failing execution. It can make an otherwise unrepeatable incident inspectable.

FoundationDB’s architecture paper describes simulation as a central development practice. The lesson is the value of repeatable fault exploration, not that simulation automatically covers hardware or operational behavior absent from its model. FoundationDB paper

The model must be reviewed like any other implementation. If it treats disk flush as perfectly instantaneous or never permits a particular timeout race, it cannot expose bugs depending on those conditions. Real-system tests complement simulation by checking assumptions about operating systems, devices, and deployment.

A failing simulation seed is evidence that can become a regression scenario. The fix should address the violated invariant rather than merely alter timing so the seed stops triggering. Otherwise another schedule may reproduce the same flaw.

Alerting should protect the user objective

Google’s monitoring guidance emphasizes user-visible symptoms alongside useful internal signals. A high CPU alert alone can be noisy when the service is meeting its objective, while a low-CPU storage stall can be severe. Monitoring distributed systems

For an illustrative 99.9% availability objective over a 30-day month, allowed unavailability is 0.001 × 30 × 24 × 60 = 43.2 minutes. The definition of an unavailable operation matters: timeout, incorrect result, or failure to meet the required latency may be counted according to the service contract.

Burn-rate alerts can detect when the service consumes its allowed error budget too quickly. Long-window and short-window evidence help distinguish a brief blip from sustained failure. The exact thresholds should follow operational needs rather than be copied without context.

An alert should point toward action. Archive lag threatening recovery, free space approaching maintenance limits, and sustained deadline failure have clear consequences. Dashboards can retain broader diagnostic detail without paging someone for every fluctuation.

Evidence and interpretation remain separate

A good incident analysis distinguishes observations, hypotheses, interventions, and conclusions. “Commit waits rose after the backup started” is an observation. “Backup I/O caused the waits” is a hypothesis until supported. “Rate-limiting backup restored latency while demand stayed similar” is stronger causal evidence, with remaining confounders noted.

The same discipline applies to published comparisons. A vendor-reported benchmark can be useful evidence about its stated workload while leaving production fit uncertain. An original numerical model can explain a bottleneck without pretending to be a measurement.

Verification closes the loop between design and operation. It asks whether the running system preserves the promised outcomes, whether its resource model predicts saturation, and whether recovery works under the specified faults. That is the foundation for responsible optimization at scale.

A fault history preserves uncertainty

Table: A test history distinguishes client observations from server evidence. Missing replies remain uncertain rather than becoming assumed failures.

This structure supports both debugging and model checking. It prevents a test harness from declaring a duplicate write harmless merely because the first call timed out. It also prevents falsely accusing a system of losing a write that was never acknowledged under the measured contract.

The most useful final report combines workload conditions, exact observation boundaries, fault schedule, correctness evidence, and remaining assumptions. It makes clear what was established and what the test did not explore. That precision is essential when performance findings are used to justify a production architecture.

Design conclusion

Observability and verification close the loop between the intended contract and the running system. They need the same precise boundaries developed throughout the course. The final part applies that vocabulary to published architectures and then assembles a complete design for three workloads with different constraints.

Sources and further reading

• Google SRE: monitoring distributed systems
• FoundationDB architecture paper, 2021
• SQLite testing architecture
• Jepsen consistency models

Course index · Lesson plan