Glossary / Concurrency limits

Concurrency limits

A concurrency limit bounds how many asynchronous operations are in flight at once. It protects downstream services, connection pools, memory, rate limits, and user experience. Promise.all(items.map(work)) launches all mapped operations eagerly, which can overwhelm a system for large or unbounded collections.

Bounded worker pattern

async function mapLimit(items, limit, worker) {
  const results = new Array(items.length);
  let next = 0;
  async function run() {
    while (true) {
      const index = next++;
      if (index >= items.length) return;
      results[index] = await worker(items[index], index);
    }
  }
  await Promise.all(Array.from({ length: Math.min(limit, items.length) }, run));
  return results;
}

This pattern preserves result order while allowing at most limit workers. Production implementations must define failure policy: fail-fast, collect errors, stop scheduling new items, or continue. They should support cancellation, avoid a shared increment race if moved across agents, and account for retry delays and rate limits. Concurrency and request rate are different: long-running calls can respect concurrency while still exceeding a requests-per-second limit.

Choose limits from downstream capacity, latency, payload size, and memory—not arbitrary large constants. Add queue bounds and backpressure for streaming producers. Instrument active count, queue depth, wait time, failure rate, and saturation.

Interview questions

  1. Why not Promise.all on a huge list? It starts everything at once and can exhaust resources or exceed service limits.
  2. Concurrency versus rate limit? Number of simultaneous operations versus operations per unit time.
  3. What failure policies matter? Fail-fast, collect-all, cancel pending, or continue while recording errors.
  4. What metrics help tune a limit? In-flight count, queue wait, latency, error rate, and downstream saturation.

References