Concurrency and throughput are related, but they are not the same metric.
Quick Definitions
- Concurrency: how many requests are in-flight at the same time.
- Throughput: how many requests are completed per second.
A system can have high concurrency and low throughput if latency is high.
Useful Relationship
A practical approximation is:
throughput ≈ concurrency / latency_seconds
This helps estimate expected throughput from known concurrency and average response time.
Worked example
With 200 concurrent requests and 400 ms average service time:
throughput ≈ 200 / 0.4 = 500 RPS
If service time rises to one second while concurrency is capped at 200, the estimate falls to 200 RPS. Raising concurrency may recover throughput temporarily, but it also increases queue, memory, connection and downstream pressure.
What to Optimize First
- If saturation is CPU-bound, optimize compute and query efficiency.
- If saturation is I/O-bound, optimize downstream latency and connection strategy.
- If queueing is the issue, improve backpressure and request prioritization.
Read the Metrics Together
Concurrency describes occupancy; throughput describes completed work. Latency links them. Review all three over the same interval and split by endpoint or operation. A healthy global throughput number can hide one route whose concurrency is accumulating because a downstream dependency slowed down.
Also distinguish client concurrency from server concurrency. A load generator may have 500 virtual users while only 120 requests are in flight because users pause between iterations. Conversely, retries and asynchronous fan-out can make server-side concurrency exceed the apparent client count.
A Practical Decision Sequence
- Measure achieved throughput, in-flight requests and latency percentiles.
- Find the first saturated constraint: CPU, connection pool, worker pool, queue or dependency.
- Change one limit or bottleneck at a time.
- Repeat the same workload and compare the full time series, not only the peak.
The approximation assumes a stable interval and representative average latency. Bursts, long-tail requests, streaming responses and queued asynchronous work need a more detailed model. Use it to frame an experiment, then validate with measurements.
Tooling
Use the Concurrency Calculator and API Throughput Calculator together to validate scenarios before running full load tests.