Glossary / Streams

Streams

Streams represent data that arrives or is produced incrementally rather than as one complete value. The Streams API lets JavaScript process large, slow, or unbounded data with bounded buffering, composable transformations, cancellation, and backpressure instead of eagerly materializing every byte or item in memory.

The stream model

A readable source produces chunks; a writable sink consumes them; a transform has both sides and maps input chunks to output chunks. A stream has a controller, a queue, and a strategy that determines when it is considered full. Queuing strategies typically account by chunk count or by a caller-provided size function. Backpressure is a signal from a slower consumer to reduce or pause production.

const response = await fetch("/large.ndjson");
const reader = response.body.getReader();
try {
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    processBytes(value);
  }
} finally {
  reader.releaseLock();
}

A reader locks a readable stream while acquired. pipeTo and pipeThrough offer composition and normally propagate close, error, and cancellation according to options. tee() creates two branches but may buffer for the slower branch; it is not free duplication.

Lifecycle and error semantics

Readable streams can be readable, closed, or errored. Writable streams can be writable, closing, closed, or errored. writer.write() returns a promise that settles when the chunk is accepted according to stream queuing, not necessarily when durable storage or a remote peer has committed it. writer.ready resolves when capacity is available again. Closing signals no more writes; abort requests termination with a reason. Errors propagate through piping unless configured otherwise.

Streams are useful for network bodies, files, decompression, parsing, media, and incremental generation. They do not automatically make a workload low-memory: code can defeat streaming by collecting all chunks, using an unbounded queue, teeing to a stalled consumer, or ignoring backpressure. Cancellation and resource cleanup must reach the underlying source where possible.

Engineering guidance

Prefer standard pipe composition where it expresses ownership and error flow. Set meaningful queue strategies and handle abort/error paths. Test slow consumers, empty streams, partial chunks, cancellation, and failures at every stage. A chunk is an arbitrary delivery boundary, not necessarily a line, record, character, or protocol message; parsers must preserve partial state across chunk boundaries.

Interview questions

  1. What problem does backpressure solve? It prevents a fast producer from accumulating unbounded queued data when a consumer is slower.
  2. Is one chunk one record? No. Chunk boundaries are transport or implementation boundaries unless the source contract says otherwise.
  3. What does writer.write() success mean? The write was accepted by the stream according to its queue and sink contract; it does not universally mean durable external commit.
  4. Why can tee() increase memory? One branch may run slower, requiring queued data to be retained for it.
  5. What should be tested? Slow consumers, errors, cancellation, close, and data split across arbitrary chunk boundaries.

References