Node.js Streams

Published by Programming Line in Node.js. Last updated .

A stream in Node.js is an abstraction for working with data that arrives piece by piece over time, instead of all at once. Rather than loading an entire file, HTTP request body, or large dataset into memory, a stream lets your program process chunks as they arrive, keeping memory usage flat and letting work start before all the data even exists. Streams are not a bolt-on feature — they sit underneath much of Node’s core: file I/O, HTTP requests and responses, TCP sockets, and process.stdin/stdout are all streams. Understanding them is essential for writing Node programs that handle real-world amounts of data efficiently.

Overview: How Streams Work

Every stream in Node.js is built on top of EventEmitter. Data doesn’t just appear as a return value — it is emitted as events over time, in pieces called chunks. A chunk is usually a Buffer (raw bytes) unless you set an encoding or enable object mode, in which case chunks can be strings or arbitrary JavaScript values. There are four fundamental stream types, and almost everything else in Node is built from them.

Type Direction Examples
Readable A source you read data from fs.createReadStream(), an HTTP server’s req, process.stdin
Writable A destination you write data to fs.createWriteStream(), an HTTP server’s res, process.stdout
Duplex Both readable and writable, independently net.Socket (a TCP connection)
Transform A duplex stream that modifies data as it passes through zlib.createGzip(), a crypto cipher stream

Flowing vs. Paused Mode, and the Internal Buffer

A Readable stream has an internal buffer and can be in paused mode or flowing mode. In paused mode you must explicitly call .read() to pull data. As soon as you attach a 'data' listener (or call .pipe()), the stream switches to flowing mode and starts pushing chunks to you automatically as they become available. Internally, each stream has a highWaterMark — the maximum number of bytes (or objects, in object mode) to buffer before the stream tells its source to slow down. Generic streams default to a 16 KiB (16384 byte) highWaterMark; fs streams default to 64 KiB (65536 bytes). You can override it by passing { highWaterMark: N } to the stream constructor.

The highWaterMark is the mechanism behind backpressure: if a Writable’s internal buffer fills up faster than it can flush to disk, network, etc., writes should pause until the buffer drains. Ignoring backpressure is one of the most common sources of runaway memory growth in Node programs, and it’s covered in detail below.

Syntax

The general shape of connecting streams together, using the modern pipeline() helper, looks like this:

import { pipeline } from "node:stream/promises";

await pipeline(
  readableSource,
  transformStream, // optional, can chain multiple
  writableDestination
);
  • readableSource — any Readable stream: a file, an incoming HTTP request, a socket.
  • transformStream — zero or more Transform streams the data passes through and is modified by (optional).
  • writableDestination — any Writable stream: a file, an outgoing HTTP response, a socket.
  • pipeline() returns a Promise that resolves when the destination has finished, or rejects if any stream in the chain errors — and it automatically destroys every stream in the chain on failure, closing file descriptors for you.

The individual building blocks you’ll use constantly are summarized here:

Method / Event Belongs to Description
.on('data', cb) Readable Fires once per chunk, in flowing mode
.on('end', cb) Readable Fires once, after the last chunk, when there is no more data
.on('error', cb) all streams Fires on failure — must be handled, or the process crashes
.write(chunk) Writable Returns false when the internal buffer is full (a backpressure signal)
.on('drain', cb) Writable Fires when it’s safe to resume writing
.end([chunk]) Writable Signals that no more data will be written
.pipe(dest) Readable Connects to a Writable manually — does not forward errors
pipeline(...) node:stream/promises Connects streams, forwards errors, and cleans up — returns a Promise

Examples

Example 1: Reading a File with a Readable Stream

import { createReadStream } from "node:fs";

const readStream = createReadStream("data.txt", { encoding: "utf8", highWaterMark: 64 * 1024 });

let chunkCount = 0;

readStream.on("data", (chunk) => {
  chunkCount++;
  console.log(`Received chunk ${chunkCount}, length: ${chunk.length}`);
});

readStream.on("end", () => {
  console.log(`Finished reading. Total chunks: ${chunkCount}`);
});

readStream.on("error", (err) => {
  console.error("Read stream error:", err.message);
});

Received chunk 1, length: 118
Finished reading. Total chunks: 1

Because data.txt here is smaller than the 64 KiB highWaterMark, the whole file arrives as a single chunk. A larger file would produce multiple 'data' events, one per chunk, until 'end' fires. Passing encoding: "utf8" makes each chunk a string instead of a raw Buffer — leave it off when you want to handle binary data yourself.

Example 2: Writing Data with Backpressure Handling

import { createWriteStream } from "node:fs";

const writeStream = createWriteStream("output.txt");

writeStream.on("error", (err) => {
  console.error("Write stream error:", err.message);
});

writeStream.on("finish", () => {
  console.log("All data has been written to output.txt");
});

function writeManyLines(stream, total) {
  let i = 0;

  function writeNext() {
    let ok = true;
    while (i < total && ok) {
      i++;
      const line = `Line number ${i}\n`;
      if (i === total) {
        stream.end(line);
      } else {
        ok = stream.write(line);
      }
    }
    if (i < total) {
      stream.once("drain", writeNext);
    }
  }

  writeNext();
}

writeManyLines(writeStream, 100000);

All data has been written to output.txt

Writing 100,000 lines in a tight loop without checking .write()‘s return value would queue all of it in memory instantly. Here, stream.write(line) returns false once the internal buffer crosses its highWaterMark, so the loop stops, and writing only resumes once the 'drain' event confirms the OS has caught up. This is backpressure handled correctly, by hand.

Example 3: Transforming Data with pipeline()

import { createReadStream, createWriteStream } from "node:fs";
import { Transform } from "node:stream";
import { pipeline } from "node:stream/promises";

const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase());
  },
});

try {
  await pipeline(
    createReadStream("data.txt"),
    upperCaseTransform,
    createWriteStream("data-upper.txt")
  );
  console.log("Pipeline finished. data-upper.txt has been written.");
} catch (err) {
  console.error("Pipeline failed:", err.message);
}

Pipeline finished. data-upper.txt has been written.

A Transform stream is both Readable and Writable: its transform(chunk, encoding, callback) method receives each incoming chunk and must call callback(null, transformedChunk) to push the result downstream (or callback(err) to fail). pipeline() wires the read stream, the transform, and the write stream together, handling backpressure and error propagation across all three automatically — something a chain of .pipe() calls does not do safely.

How It Works Step by Step (Under the Hood)

Walking through Example 3’s pipeline() call, in order:

  1. The file system is not truly asynchronous at the OS level on most platforms, so createReadStream‘s reads are dispatched to libuv’s thread pool (four threads by default, configurable with UV_THREADPOOL_SIZE). A worker thread reads a chunk from disk.
  2. When the chunk is ready, libuv schedules the completion callback onto the event loop’s poll phase, and the Readable stream emits it internally.
  3. The chunk is written into upperCaseTransform. Its transform() method runs, uppercases the text, and calls callback(null, result), which pushes the result into the Transform’s own internal readable buffer.
  4. The Transform emits that chunk downstream, and createWriteStream("data-upper.txt") writes it to disk (again via the thread pool).
  5. If the write stream’s buffer fills up, its .write() returns false; pipeline() pauses the upstream source until 'drain' fires, so memory never grows unbounded regardless of file size.
  6. Once the source emits 'end', the transform is ended, the destination emits 'finish', and the Promise returned by pipeline() resolves.
  7. If an error occurs at any stage — a missing file, a full disk — pipeline() destroys every stream in the chain and rejects the promise, which is why a single try/catch around it is enough to handle failures from the whole chain.

By contrast, network streams like http and net don’t need the thread pool at all — the OS itself delivers socket readiness asynchronously, so those streams read and write directly through the event loop’s poll phase.

Common Mistakes

Mistake 1: Using .pipe() Without Error Handling

Wrong:

import { createReadStream, createWriteStream } from "node:fs";

createReadStream("big-file.txt").pipe(createWriteStream("copy.txt"));

If big-file.txt doesn’t exist, the Readable stream emits an 'error' event. Nothing is listening for it, so Node throws an uncaught exception and the whole process crashes — and .pipe() never closes the destination stream on that failure, potentially leaving a partial file and a dangling file descriptor. Corrected, using pipeline():

import { createReadStream, createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";

try {
  await pipeline(createReadStream("big-file.txt"), createWriteStream("copy.txt"));
  console.log("Copy complete.");
} catch (err) {
  console.error("Copy failed:", err.message);
}

Mistake 2: Reading a Whole File into Memory Instead of Streaming

Wrong:

import { readFileSync } from "node:fs";

function handleUpload(req, res) {
  const fileData = readFileSync("huge-video.mp4");
  res.end(fileData);
}

Inside a request handler, readFileSync is doubly wrong: it blocks the entire event loop until the read finishes, freezing every other request the server is handling, and it loads the complete file — potentially gigabytes — into memory at once. Corrected, using a stream:

import { createReadStream } from "node:fs";

function handleUpload(req, res) {
  const stream = createReadStream("huge-video.mp4");
  stream.on("error", (err) => {
    res.statusCode = 500;
    res.end("Failed to read file");
  });
  stream.pipe(res);
}

This streams the file to the client in small, bounded chunks, never blocks the event loop, and reports a clean 500 response instead of crashing if the file can’t be read.

Best Practices

  • Prefer pipeline() from node:stream/promises over manual .pipe() chains — it forwards errors and cleans up every stream automatically.
  • Always attach an 'error' listener to every stream you create; an unhandled stream error crashes the Node process.
  • Avoid *Sync file methods (like readFileSync) inside request handlers or any other hot path — use streams or the fs/promises API instead.
  • Respect the boolean returned by .write(); pause writing and wait for 'drain' before writing more, or let pipeline()/.pipe() manage it for you.
  • Tune highWaterMark to your data: too low adds overhead from frequent small chunks, too high increases peak memory use.
  • Use { objectMode: true } when streaming JavaScript values (e.g. database rows) instead of raw bytes.
  • Call .destroy() on a stream you no longer need, to release its file descriptor or socket promptly.
  • Reach for built-in Transform streams — zlib.createGzip(), crypto ciphers — instead of buffering data fully and transforming it afterward.

Practice Exercises

  1. Write a program that uses createReadStream and a Transform stream to count the number of lines in a large text file without ever loading the whole file into memory. Hint: count the occurrences of \n in each incoming chunk.
  2. Build a program that copies a file using pipeline(), adding a second Transform stage in the middle that compresses the data with node:zlib‘s createGzip(). Confirm the resulting .gz file can be decompressed back to the original content.
  3. Write an HTTP server with node:http that streams a local file back to the client with createReadStream().pipe(res), and correctly responds with a 404 status — without crashing the server — when the requested file doesn’t exist.

Summary

  • Streams process data in chunks over time instead of loading everything into memory at once, keeping memory usage flat regardless of input size.
  • Node has four stream types — Readable, Writable, Duplex, and Transform — all built on EventEmitter.
  • Backpressure (the 'drain' event and .write()‘s return value) protects your program from unbounded memory growth when a destination is slower than its source.
  • pipeline() from node:stream/promises is the modern, safe way to connect streams: it forwards errors and cleans up automatically, unlike raw .pipe() chains.
  • Always attach an 'error' listener — an unhandled stream error crashes the entire Node process.
  • Use streaming APIs instead of *Sync file reads whenever handling requests or working with large files.