Glossary / Node.js streams

Node.js streams

Node.js streams are an event-driven abstraction for handling data incrementally, including files, sockets, HTTP bodies, and transforms. They let applications process large or ongoing data without buffering it all at once, and provide backpressure so producers can respond to slower consumers.

Stream categories and flow

A Readable produces data; a Writable consumes it; a Duplex does both independently; a Transform is a duplex whose output derives from input. Streams can operate in flowing mode (data events) or paused/readable mode (explicit reads). Prefer the pipeline() utility for connecting streams: it forwards errors, handles cleanup, and supports completion callbacks or promises.

import { createReadStream, createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';
await pipeline(
  createReadStream('source.bin'),
  transform,
  createWriteStream('destination.bin'),
  { signal: abortSignal }
);

For writable streams, write(chunk) returning false means the internal buffer reached its high-water mark; stop writing until drain. Continuing to write ignores backpressure and can cause memory growth. highWaterMark is a threshold, not a strict maximum: one large chunk can exceed it, and upstream work may already be in flight.

Errors, destruction, and interop

A stream can emit error; unhandled errors can crash a process. end means readable input ended; finish means writable output finished; close indicates underlying resources closed and is not synonymous with successful completion. destroy(error) tears down a stream. Use pipeline or finished to observe completion. Mixing event mode, async iteration, and .pipe() ownership can create confusing flow behavior.

Node also exposes WHATWG Web Streams and interop adapters; their APIs and lifecycle semantics differ in details. Be clear which stream family a function consumes, and adapt at boundaries.

Engineering guidance

Prefer pipeline over hand-wiring multiple .pipe() calls when error cleanup matters. Respect write() return values, use async iteration for sequential consumption, and configure high-water marks based on memory and latency goals. Test errors in every stage, early cancellation, slow sinks, and partial chunks.

Interview questions

  1. What does write() returning false mean? Stop producing until drain indicates capacity is available.
  2. Why use pipeline()? It propagates errors and handles teardown across stages.
  3. Does highWaterMark cap memory absolutely? No; it is a threshold and does not account for every in-flight or oversized chunk.
  4. end, finish, close? Readable completion, writable completion, and resource closure respectively.
  5. Why not mix consumption styles casually? They alter flow/ownership and can lead to unexpected buffering or missed data.

References