Find out where a slow commit waits

The shop’s purchases become slow. A dashboard shows low CPU use. That observation rules out neither lock waits nor storage waits. We need evidence tied to the request’s actual dependencies.

Metrics summarize events or resource state over time. A trace follows one operation across its steps. Profiling investigates where execution resources are spent. Verification checks whether outcomes satisfy the promised behavior, including under failures. They answer different questions.

Trace one purchase

Assume the measured request spends 2 ms executing, 18 ms waiting for a durable-log acknowledgement, and 1 ms returning the result, with no overlap in this example.

Execution
2 ms
Reads, checks, and tentative changes

Durability wait
18 ms
Waiting for the required log persistence

Reply
1 ms
Return the committed outcome

The total is 21 ms. Low CPU use is compatible with a request dominated by storage waiting.

The trace suggests where to investigate, not a proven root cause. Check storage queueing, flush duration, and competing work. A controlled experiment that changes one factor can test the explanation. Optimizing a filter based only on low overall throughput would be guesswork.

Define success before counting it

A service-level indicator is a measured property such as successful purchase rate or response time. A service-level objective sets a target over a stated window. Count failures and timeouts according to the user-facing operation, rather than reporting only the fast successes.

For correctness, retain histories containing operation IDs, invocation times, responses, and relevant observed values. A timed-out write has an uncertain outcome until evidence resolves it; it should not automatically be classified as an aborted write.

Test the persistence boundary

The logging lesson says a confirmed commit must be recoverable under its stated failure assumptions. A useful test interrupts execution at different points around log persistence, data installation, and reply delivery. After restart, compare the recovered state with the allowed outcomes.

Also test failure during recovery, malformed input, exhausted memory, and I/O errors. A test harness that merely starts the process again misses whether the right records survived. Keep a simple reference model where possible, and compare the implementation’s observable behavior with it.

Change the symptom

The next incident has 18 ms of lock waiting and only 1 ms of log waiting. Should the previous storage fix be applied automatically?

Follow the new evidence

No. The same total-latency symptom can have a different cause. Inspect the lock holder, transaction duration, and competing work. The earlier explanation was conditional on the observed path, not a permanent property of all slow commits.

A dependable engine needs evidence for both speed and correctness. The measurements tell us where work goes; adversarial tests tell us whether failures preserve the contract.

SQLite’s testing account describes concrete failure-testing categories. Google’s monitoring workbook discusses operational evidence and its use.

Extended reference

The extended chapter preserves the original technical coverage, additional examples, and citations. The explanation above supplies the concepts needed for this page; the reference develops the specialist cases.

Find a section from the original chapter