Glossary / Stream backpressure

Stream backpressure

Backpressure is the flow-control mechanism by which a consumer that cannot keep up signals upstream producers to slow down or stop. In JavaScript streams, queue sizes, high-water marks, writer.ready, source pull, and pipe propagation coordinate that pressure so data does not accumulate without bound.

How pressure is represented

A queuing strategy estimates queue cost, either by chunk count or a size(chunk) function. Its high-water mark defines the desired queue capacity. For writable streams, desiredSize reports remaining capacity and ready settles when writing can safely resume. For readable streams, the source's pull is invoked when the queue needs more data. Piping connects demand and pressure across stages.

const writer = writable.getWriter();
for (const chunk of sourceChunks) {
  await writer.ready;
  await writer.write(chunk);
}
await writer.close();

This pattern bounds the producer's outstanding work only if the producer itself does not create an unbounded number of pending tasks or retain all chunks elsewhere. A count-based strategy treats a tiny object and a huge byte buffer as equal; use a size function when byte cost is the meaningful resource.

Where backpressure gets lost

Common failures include calling write() repeatedly without awaiting or observing readiness, collecting the entire readable stream into an array, spawning unlimited asynchronous transformations, or teeing to a slow branch. Adapters to callback-based APIs may need explicit pause/resume handling. Network and operating-system buffers may have their own capacity; a stream's successful write is not necessarily remote processing or durable persistence.

Backpressure is distinct from throttling. Backpressure is demand-driven flow control between connected stages; throttling limits rate over time, often for an external quota or UI event stream. Both may be needed.

Engineering guidance

Choose queue units that approximate real memory or work, instrument queue depth and wait time, and define behavior for slow or stuck consumers. Add bounded concurrency only where parallelism is valuable, then preserve ordering if required. Test with deliberately slow sinks and large chunks; a fast local fixture often hides unbounded buffering.

Interview questions

  1. What problem does backpressure prevent? Producers outrunning consumers and building unbounded queues.
  2. How does desiredSize help? It indicates remaining writable queue capacity; nonpositive values indicate pressure.
  3. Why is write() without awaiting dangerous? It can enqueue work faster than the sink drains and hide errors.
  4. How does it differ from throttling? Backpressure follows consumer demand; throttling imposes a time-based rate limit.
  5. How can teeing affect pressure? A slow branch may force data retention while the faster branch continues.

References