Readable and Writable Streams

Streams are one of Node.js’s core building blocks for working with data that arrives piece by piece instead of all at once — files, network requests, process input, and compressed archives all flow through streams. Instead of waiting for an entire file or response to load into memory, a stream lets you process each chunk as soon as it arrives, keeping memory use flat and letting your program start working before the source has even finished sending data. Node.js models the two directions of this flow as Readable streams (data flowing out of a source, into your code) and Writable streams (data flowing from your code into a destination). Understanding how they buffer data, emit events, and apply backpressure is essential for writing Node programs that stay fast and don’t fall over under load.

Overview: How Streams Work

Every stream in Node.js is built on top of EventEmitter. A Readable stream emits a data event each time a new chunk is available, and an end event once there is no more data. A Writable stream exposes a write() method to push data out and emits a finish event once everything has been flushed to the underlying resource (a file, a socket, a compression algorithm). Both directions emit an error event when something goes wrong.

Internally, every stream keeps an internal buffer whose size is controlled by the highWaterMark option (64 KB by default for non-object streams). For a Readable stream, this buffer fills up from the underlying source (disk, socket, another process) and drains as your code consumes it. For a Writable stream, it fills up as your code calls write() and drains as the underlying resource accepts the data. Chunks are delivered as Buffer objects by default — raw bytes — unless you set an encoding option, in which case Node decodes each chunk into a string (usually utf8) before handing it to you.

A Readable stream has two reading modes. In flowing mode, data is pushed to you automatically as soon as it’s available — this happens once you attach a data listener or call .pipe(). In paused mode, you must explicitly call .read() to pull a chunk; this is the default state and gives you manual control over pacing. Most application code uses flowing mode via .pipe() or an async iterator (for await (const chunk of readable)), letting Node handle the pacing for you.

Node.js actually has four stream base classes: Readable, Writable, Duplex (both readable and writable, like a TCP socket), and Transform (a duplex stream that modifies data as it passes through, like zlib.createGzip()). This lesson focuses on plain Readable and Writable streams; Transform streams get their own lesson later in this section.

Syntax

The most common way to get a stream is through a built-in factory function rather than constructing the class directly:

const readable = fs.createReadStream(path, options);
const writable = fs.createWriteStream(path, options);

Option	Applies to	Meaning
encoding	both	Decode/encode chunks as strings (e.g. "utf8") instead of raw Buffers
highWaterMark	both	Internal buffer size in bytes before backpressure kicks in (default 64 KB)
flags	writable	File system flags, e.g. "a" to append instead of overwrite
start / end	readable	Byte offsets to read only part of a file

The events you’ll use most often:

Event	Stream type	Fires when
data	Readable	A new chunk is available (switches the stream to flowing mode)
end	Readable	No more data will ever be emitted
drain	Writable	The internal buffer has emptied enough to accept more writes
finish	Writable	All data has been flushed after calling .end()
error	both	Something failed — always listen for this
close	both	The underlying resource (file descriptor, socket) has been released

Examples

Example 1: Reading a file with a Readable stream

const fs = require("node:fs");

const readStream = fs.createReadStream("report.csv", {
  encoding: "utf8",
  highWaterMark: 64 * 1024,
});

readStream.on("data", (chunk) => {
  console.log(`Received ${chunk.length} characters`);
});

readStream.on("end", () => {
  console.log("Finished reading the file");
});

readStream.on("error", (err) => {
  console.error("Read error:", err.message);
});

Received 65536 characters
Received 12045 characters
Finished reading the file

Attaching a data listener switches the stream into flowing mode. Node reads the file in chunks up to highWaterMark bytes at a time, decodes each chunk to a UTF-8 string because encoding was set, and emits it. The exact number and size of chunks depends on the file size and the OS’s I/O behaviour — never assume a file arrives as a single chunk.

Example 2: Writing data with a Writable stream

const fs = require("node:fs");

const writeStream = fs.createWriteStream("output.txt");

writeStream.on("error", (err) => {
  console.error("Write error:", err.message);
});

writeStream.on("finish", () => {
  console.log("All data has been written");
});

writeStream.write("First line\n");
writeStream.write("Second line\n");
writeStream.end("Last line\n");

All data has been written

Each call to write() queues data to be flushed to disk; end() writes one final chunk and signals that no more writes will happen. The finish event fires only after every queued chunk has actually been flushed to the underlying file descriptor, not as soon as end() is called.

Example 3: Piping a Readable into a Writable (with compression)

const { pipeline } = require("node:stream/promises");
const fs = require("node:fs");
const zlib = require("node:zlib");

async function copyAndCompress(source, destination) {
  await pipeline(
    fs.createReadStream(source),
    zlib.createGzip(),
    fs.createWriteStream(destination)
  );
  console.log(`Compressed ${source} into ${destination}`);
}

copyAndCompress("report.csv", "report.csv.gz").catch((err) => {
  console.error("Pipeline failed:", err.message);
});

Compressed report.csv into report.csv.gz

This chains a Readable, a Transform (zlib.createGzip()), and a Writable into one pipeline. pipeline() from node:stream/promises connects them, forwards backpressure through the whole chain automatically, and — critically — propagates an error from any stage to a single rejected promise, which is exactly what plain .pipe() chains fail to do.

Example 4: Building a custom Readable stream

const { Readable } = require("node:stream");

class NumberStream extends Readable {
  constructor(max, options) {
    super(options);
    this.current = 1;
    this.max = max;
  }

  _read() {
    if (this.current > this.max) {
      this.push(null);
      return;
    }
    this.push(`${this.current}\n`);
    this.current++;
  }
}

const numbers = new NumberStream(5);
numbers.on("data", (chunk) => process.stdout.write(chunk.toString()));
numbers.on("end", () => console.log("Done generating numbers"));

1
2
3
4
5
Done generating numbers

Subclassing Readable means implementing _read(), which Node calls whenever it needs more data. You supply data by calling this.push(chunk); calling this.push(null) signals the end of the stream, which triggers the end event once all pushed data has been consumed. This pattern is how you’d stream data generated on the fly — from a database cursor, a generator function, or an API paginator — without loading it all into an array first.

Under the Hood: Backpressure and Buffering

Backpressure is what keeps a fast producer from overwhelming a slow consumer. When you call writeStream.write(chunk), it returns a boolean: true means the internal buffer is still below highWaterMark and you can keep writing; false means the buffer is full and you should stop until the stream emits a drain event. Ignoring this return value and writing as fast as possible defeats the entire purpose of streaming — the internal buffer grows unbounded and you’re back to holding everything in memory.

const fs = require("node:fs");

function writeNumbers(writeStream, total, current = 0) {
  let ok = true;
  while (current < total && ok) {
    current++;
    const line = `line ${current}\n`;
    if (current === total) {
      writeStream.write(line, () => console.log("Finished writing"));
    } else {
      ok = writeStream.write(line);
    }
  }

  if (current < total) {
    writeStream.once("drain", () => writeNumbers(writeStream, total, current));
  }
}

const out = fs.createWriteStream("numbers.txt");
writeNumbers(out, 500000);

Finished writing

Step by step: the loop writes as long as write() keeps returning true. The moment it returns false, the loop stops and registers a one-time drain listener instead of continuing to buffer more data in memory; when the underlying file catches up, drain fires and the function resumes exactly where it left off. This is precisely what .pipe() and pipeline() do internally on your behalf, which is why piping is almost always preferable to writing this loop by hand.

Common Mistakes

Mistake 1: Loading a large file into memory instead of streaming it

const fs = require("node:fs");

const data = fs.readFileSync("huge-video.mp4");
fs.writeFileSync("copy.mp4", data);

This reads the entire file into a single Buffer in memory before writing any of it back out. For a multi-gigabyte file this can exhaust available memory, and readFileSync/writeFileSync both block the event loop for the entire operation — no other request can be handled in the meantime. Stream it instead:

const fs = require("node:fs");
const { pipeline } = require("node:stream/promises");

async function copyLargeFile(source, destination) {
  await pipeline(fs.createReadStream(source), fs.createWriteStream(destination));
  console.log("Copy complete without loading the whole file into memory");
}

copyLargeFile("huge-video.mp4", "copy.mp4").catch((err) => {
  console.error("Copy failed:", err.message);
});

Now the file moves through in 64 KB chunks, memory usage stays flat regardless of file size, and the event loop is never blocked because the reads and writes are asynchronous.

Mistake 2: Forgetting to handle the error event

const fs = require("node:fs");

const readStream = fs.createReadStream("maybe-missing.txt");
readStream.pipe(fs.createWriteStream("copy.txt"));

If maybe-missing.txt doesn’t exist, the read stream emits an error event. Because streams are EventEmitters, an error event with no listener attached throws and crashes the entire Node process — not just this operation. Always attach error handlers on every stream involved:

const fs = require("node:fs");

const readStream = fs.createReadStream("maybe-missing.txt");
const writeStream = fs.createWriteStream("copy.txt");

readStream.on("error", (err) => {
  console.error("Read failed:", err.message);
});

writeStream.on("error", (err) => {
  console.error("Write failed:", err.message);
});

readStream.pipe(writeStream);

Better still, prefer pipeline() over raw .pipe() chains: it attaches error handling and cleanup (destroying every stream in the chain) automatically, so a failure partway through can’t leave a dangling open file descriptor.

Best Practices

• Prefer pipeline() from node:stream/promises over manual .pipe() chains — it forwards errors and cleans up every stream automatically.
• Always attach an error listener to every stream you create, or route it through pipeline()‘s error handling.
• Respect the return value of write(); pause on false and resume on drain instead of buffering more data yourself.
• Use for await (const chunk of readable) for simple sequential consumption when you don’t need the full power of pipe().
• Set highWaterMark deliberately for very large or very small chunk workloads instead of relying on the 64 KB default.
• Never use synchronous fs calls (readFileSync, writeFileSync) inside request handlers — they block the entire event loop.
• Call .destroy(err) on a stream you’re abandoning early to release its underlying file descriptor or socket.
• Use encoding: "utf8" only when you actually want strings; leave chunks as raw Buffers when passing them to another binary-aware stream like zlib.

Practice Exercises

• Write a script that uses fs.createReadStream and a data listener to count the total number of lines in a text file without ever holding the whole file in memory at once. (Hint: count newline characters per chunk.)
• Build a custom Writable stream (subclass Writable and implement _write(chunk, encoding, callback)) that uppercases every chunk it receives and prints it to the console.
• Use pipeline() to chain fs.createReadStream, zlib.createGunzip(), and fs.createWriteStream to decompress a .gz file, then verify the pipeline correctly reports an error when you point it at a source file that doesn’t exist.

Summary

• A Readable stream emits data out of a source in chunks; a Writable stream accepts data into a destination in chunks.
• Both are built on EventEmitter: key events are data, end, drain, finish, and error.
• An internal buffer sized by highWaterMark controls how much data is held in memory before backpressure applies.
• write() returns false when the buffer is full — stop writing and wait for drain instead of ignoring it.
• Prefer pipeline() over manual .pipe() chains for correct error propagation and cleanup.
• Streaming keeps memory flat and avoids blocking the event loop, which is why it beats readFileSync/writeFileSync for large data.
• You can build custom streams by subclassing Readable (implement _read) or Writable (implement _write).