Latency, Throughput, Capacity, and Bottlenecks

The short answer

Latency is the time one operation takes. Throughput is work completed per unit time. Capacity is the workload a system can sustain while meeting defined quality goals. A bottleneck is the limiting part of that workload. These quantities interact, but they are not interchangeable.

The previous lesson asked what the system needs to do. This one asks how to measure whether its current design can do it.

Latency

End-to-end latency is the elapsed time from a chosen start to a chosen finish. State those points: a server-side timer from request arrival to response is not the same as a user's time from click to visible update. Network, application, database, queueing, and browser work can contribute, sometimes overlapping. Google SRE notes that client-side latency can better reflect user experience even when only server-side data is easy to collect.Do not add separately measured stage times as if they always form a neat, non-overlapping sum.¹

Throughput

Throughput is a rate of completed work: HTTP requests per second, jobs per second, messages per minute, or bytes per second. Always name the operation and unit. “500 per second” is ambiguous; a lightweight read and a costly report do not consume equal resources. Google SRE's overload discussion warns that even requests per second can be a poor capacity proxy when request costs vary.Count successful and failed work separately where that distinction matters.³

Concurrency

Concurrency is how many operations are active or waiting at once. It is not the same as concurrent users or requests per second. A logged-in user can be idle. A long-lived connection can remain open without generating many requests. If more operations arrive while workers or database connections remain busy, waiting can increase even when arrival rate barely changes.

Average versus percentile latency

The mean adds observations and divides by their count. The median, or p50, splits an ordered set roughly in half. A p95 threshold means about 95% of observations are at or below it under the chosen calculation method and window; p99 looks farther into the slow tail. The exact percentile computation can vary for small samples, so do not over-interpret a ten-request toy set.

Illustrative observations: nine requests take 100 ms each and one takes 1,000 ms. The mean is (9 × 100 ms + 1 × 1,000 ms) ÷ 10 = 190 ms; the median is 100 ms. Neither number alone tells the whole story. Google SRE recommends examining latency distributions because averages can hide slow requests.Choose percentiles based on user needs, not because p95 or p99 sounds impressive.¹

Capacity

Capacity is conditional: how much of which operation can this system sustain while meeting which latency and error goals? A server that completes more requests by letting most users wait too long has not necessarily met the product need. Capacity also changes with data size, query mix, configuration, and dependencies.

Peak and average traffic

A daily average smooths out bursts. Google SRE gives an example in which alternating seconds at 200 and zero requests per second have the same average as a steady 100 per second, despite a different instantaneous demand.Measure or estimate peaks over a relevant window; a minute-wide average can conceal a five-second spike.¹

Back-of-the-envelope estimation

Use fictional planning inputs, not a production benchmark:

• 100,000 active users/day × 20 requests/user/day = 2,000,000 requests/day.
• 2,000,000 requests/day ÷ 86,400 seconds/day ≈ 23 requests/second as a 24-hour average.
• With an assumed 10× peak-to-average factor, 23.15 requests/second × 10 ≈ 231 requests/second during a modeled peak.

The exact unrounded average is about 23.15 requests/second; rounding at the end gives about 231 for the peak. The factor is invented for this exercise, not observed. This estimate excludes bots, retries, background jobs, caches, failed requests, and differences in request cost. Real traffic is not uniformly spread through a day. Use measured workload before buying capacity.

Bottlenecks

The limiting resource may be CPU, memory, disk I/O, network bandwidth, a lock, a worker pool, database connections, a slow query, an external service, or browser work. Identify the constrained operation and time window. Adding application servers does not fix a database connection limit; it may increase pressure on that limit.

Queueing at a high level

When arrivals temporarily outpace available workers or connections, some operations wait. Waiting adds latency even if the work itself takes the same time. As utilization rises, small bursts can create a disproportionately large queue. This is a qualitative model, not a guarantee that every system follows one formula. The later resilience lesson will cover overload controls.

Resource utilization

CPU at 50% does not mean the entire system has 50% free capacity. One dependency may be saturated while another is idle. Google SRE calls saturation one of four useful monitoring signals, alongside latency, traffic, and errors.Look for wait time, queue depth, pool occupancy, and error rates alongside headline utilization.²

Measuring before optimizing

Record end-to-end timing and a few meaningful internal boundaries, then compare quiet and peak periods. A load test is useful for exploring possible limits, but it is not a substitute for observing production traffic and real request mixes. Improving a non-bottleneck may make a local metric prettier without changing user latency. After a fix, measure again: the bottleneck can move.

Common misconceptions

Capacity exercise

Repeat the fictional estimate with 5 requests/user/day rather than 20. At the same 100,000 users/day, the average is 500,000 ÷ 86,400 ≈ 5.79 requests/second; the assumed 10× peak is about 57.9 requests/second. The answer changed because the workload assumption changed, not because the architecture changed.

Debugging scenario

Why this matters when reviewing AI-generated designs

Generated recommendations may quote throughput without units, assume uniform traffic, size machines without a workload, or optimize average latency while the slow tail remains unacceptable. Require the input assumptions, formula, measurement window, quality target, and evidence of the actual bottleneck. Treat an unsupported “this handles one million users” claim as a question, not a fact.

Knowledge check

What to learn next