Separate time working from time waiting

A product lookup performs 5 ms of useful processing but takes 50 ms to answer. Calling the query “a 5 ms operation” ignores most of the user’s wait. We need a timeline before we need an optimization.

Response time runs from the chosen start boundary to completion. Service time is time spent doing the relevant work. Queueing time is time waiting for permission or capacity to do it. Other waits, such as storage or communication, must be identified rather than hidden inside an unexplained total.

Put the intervals on one request

Assume a simplified sequential path with no overlap:

0–30 ms: queued
The request waits for a worker.
30–35 ms: processing
The worker finds the requested row.
35–50 ms: remaining wait and delivery
The result becomes available to the caller at 50 ms.

The caller experiences 50 ms. Optimizing the 5 ms processing interval alone cannot remove the other 45 ms.

If processing becomes twice as fast, the total becomes 47.5 ms under these unchanged assumptions. That is a 5% reduction, not a twofold end-to-end speedup. In a real system, faster processing can also reduce queueing, so measurements may differ; the point is to identify the causal path.

Count work at the same boundary

Throughput is completions per unit time. Concurrency is work currently in progress within a chosen boundary. Under stable long-run conditions, Little’s Law relates their averages: average work in the system equals average completion rate times average time in the system.

At 200 completed requests per second and 0.05 seconds average response time, the corresponding average is 10 requests in progress. This is not a recommended thread count. Some of that work may be queued or waiting rather than executing.

Do not mix successful purchases per second with latency measured for individual retry attempts. They count different units of work.

Make the load generator match the question

A test that sends a request only after the previous reply slows its own arrivals when the server slows. That can hide the queue that independent arriving users would create. Report how requests are scheduled, including timeouts, rejected work, and unfinished work.

An average also hides variation. A percentile describes a position in the observed distribution: a 99th-percentile latency is a threshold reached by roughly 99% of samples under the chosen calculation. It does not describe the worst request or every user’s experience.

Change only the processing interval

If processing drops from 5 ms to 1 ms while the other intervals remain unchanged, what is the new response time?

Add the whole path

46 ms: 30 + 1 + 15. The local processing improvement is fivefold, but the end-to-end reduction is 4/50, or 8%. This calculation uses fixed non-overlapping intervals; it is a model to compare with measurements, not a forecast for every workload.

Now we can distinguish a slow operation from an overloaded service. Next we follow what happens when arrivals exceed the rate of useful completions.

Google’s monitoring chapter discusses latency, traffic, errors, and saturation as different observations. The extended reference derives the queueing relationships in more depth.

Extended reference

The extended chapter preserves the original technical coverage, additional examples, and citations. The explanation above supplies the concepts needed for this page; the reference develops the specialist cases.

Find a section from the original chapter