Node.js Buffers

A Buffer is a Node.js built-in class for working with raw, fixed-length binary data — the kind of data that comes from files, network sockets, and other I/O sources before it has any particular text encoding applied to it. JavaScript in the browser never needed this because it rarely touched raw bytes, but a server-side runtime constantly reads files, receives TCP packets, and streams binary payloads, so Node added Buffer as a global class available in every module without an import. Understanding buffers is essential for working with the fs, net, http, and stream modules correctly and efficiently.

Overview: How Buffers Work

A Buffer represents a fixed-length sequence of bytes (integers from 0 to 255). Internally, Buffer is a subclass of JavaScript’s Uint8Array, so every typed-array method (slice, indexOf, iteration, .length) works on it, but Node adds extra methods on top for encoding conversion, comparison, and I/O convenience. Unlike a JavaScript string, a buffer has no character encoding — it is just bytes. You choose an encoding (like utf8, hex, or base64) only when you convert between a buffer and a string.

Buffers matter because strings in JavaScript are stored internally as UTF-16, which is not the same layout as the bytes arriving over a network socket or sitting in a JPEG file. If Node tried to represent every incoming chunk of file or socket data as a string immediately, it would have to guess an encoding, and binary formats like images or compressed archives are not valid text at all. Buffers let Node hand you the bytes exactly as they are, and you decide how (or whether) to interpret them as text.

The Buffer Pool

For performance, Node pre-allocates a shared internal memory pool (Buffer.poolSize, 8 KB by default) for small buffers. When you call Buffer.allocUnsafe(size) with a size at or under half the pool size, Node slices a piece out of this existing pool instead of asking the OS for new memory — this is much faster, but the memory is not zeroed out, so it may contain leftover data from something else that used that memory earlier. Buffer.alloc(size) is the safe counterpart: it always zero-fills the memory before handing it to you, at a small performance cost. Buffers created this way are still normal JavaScript objects from the garbage collector’s point of view — Node handles their memory lifecycle for you, you never manually free a buffer.

Syntax

The most common ways to create and convert buffers:

  • Buffer.alloc(size[, fill[, encoding]]) — allocates a zero-filled buffer of size bytes (optionally filled with a value).
  • Buffer.allocUnsafe(size) — allocates a buffer of size bytes without zeroing it; faster but may contain old memory contents until you overwrite it.
  • Buffer.from(source[, encoding]) — creates a buffer from a string, array of bytes, ArrayBuffer, or another buffer.
  • buf.toString([encoding[, start[, end]]]) — decodes some or all of a buffer’s bytes back into a string using the given encoding (default utf8).
  • buf.write(string[, offset[, length]][, encoding]) — writes a string into an existing buffer at a given offset.
  • Buffer.concat(listOfBuffers[, totalLength]) — joins an array of buffers into one new buffer.
  • Buffer.byteLength(string[, encoding]) — returns the number of bytes a string would take up when encoded, which can differ from string.length.
  • Buffer.compare(buf1, buf2) / buf1.equals(buf2) — compare buffer contents byte by byte.
  • buf.subarray(start, end) — returns a view over the same underlying memory (no copy), the modern replacement for the older buf.slice().
Encoding Description
utf8 Default. Variable-width Unicode text (1–4 bytes per character).
utf16le / ucs2 2 or 4 bytes per character, little-endian.
latin1 1 byte per character, ISO-8859-1 range only.
ascii 7-bit ASCII; strips the high bit of each byte.
base64 / base64url Text-safe binary encoding, commonly used for tokens and data URLs.
hex Each byte represented as two hexadecimal characters.

Examples

Example 1: Creating and Converting Buffers

const buf1 = Buffer.alloc(10);              // 10 zero-filled bytes
const buf2 = Buffer.alloc(10, 1);            // 10 bytes filled with the value 1
const buf3 = Buffer.from("Hello");           // from a UTF-8 string
const buf4 = Buffer.from([72, 101, 108, 108, 111]); // from raw byte values

console.log(buf1);
console.log(buf2);
console.log(buf3);
console.log(buf3.toString());
console.log(buf3.toString("hex"));
console.log(buf3.toString("base64"));
console.log(buf3.equals(buf4));

Output:

<Buffer 00 00 00 00 00 00 00 00 00 00>
<Buffer 01 01 01 01 01 01 01 01 01 01>
<Buffer 48 65 6c 6c 6f>
Hello
48656c6c6f
SGVsbG8=
true

Notice how console.log prints a buffer as a list of hexadecimal byte values inside angle brackets — that is Node’s default inspection format, not the buffer’s content as text. Calling .toString() is what decodes those bytes into a readable string, and the encoding you pass controls how they are decoded.

Example 2: Byte Length vs String Length

const str = "caf\u00e9 \u2615"; // "café ☕"

console.log("string length:", str.length);
console.log("byte length (utf8):", Buffer.byteLength(str, "utf8"));

const part1 = Buffer.from("Hello, ");
const part2 = Buffer.from("world!");
const combined = Buffer.concat([part1, part2]);

console.log(combined.toString());
console.log("combined byte length:", combined.length);

Output:

string length: 6
byte length (utf8): 9
Hello, world!
combined byte length: 13

The accented é and the ☕ emoji each take more than one byte in UTF-8, even though they count as a single character in a JavaScript string. This mismatch matters whenever you allocate a buffer sized to hold a string — always size it with Buffer.byteLength(), never string.length. Buffer.concat() also shows the correct way to join buffers: it returns one new buffer containing the combined bytes, unlike the + operator.

Example 3: Reading a Binary File and Checking Its Type

const fs = require("node:fs/promises");
const path = require("node:path");

const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);

async function inspectFile(filePath) {
  const data = await fs.readFile(filePath);
  const header = data.subarray(0, PNG_SIGNATURE.length);

  if (Buffer.compare(header, PNG_SIGNATURE) === 0) {
    console.log(`${path.basename(filePath)} is a PNG image`);
  } else {
    console.log(`${path.basename(filePath)} is not a PNG image`);
  }

  console.log(`File size: ${data.length} bytes`);
  console.log(`Base64 preview: ${data.toString("base64").slice(0, 40)}...`);
}

async function main() {
  try {
    await inspectFile(process.argv[2] ?? "./example.png");
  } catch (err) {
    console.error("Failed to read file:", err.message);
  }
}

main();

Output (for a real PNG file):

example.png is a PNG image
File size: 4821 bytes
Base64 preview: iVBORw0KGgoAAAANSUhEUgAAAyAAAAGQCAYAAAB...

This reads the entire file into a single Buffer with fs.readFile (using the promise API, no encoding argument so it stays binary). Every file format has a "magic number" — fixed leading bytes that identify its type — and PNG’s is the classic example. Comparing raw bytes with Buffer.compare() is exactly how real file-type detection libraries work under the hood.

How It Works Step by Step

When you call Buffer.allocUnsafe(200), Node checks whether 200 bytes fit within its current shared pool. If they do, it returns a view into existing memory (no zeroing, no new OS allocation) and advances the pool’s internal offset. If the pool doesn’t have enough room left, or the requested size exceeds half of Buffer.poolSize, Node allocates a dedicated block of memory outside the pool instead. Either way, the resulting Buffer object is a normal JavaScript object that the V8 garbage collector tracks and frees automatically once nothing references it — you never call a "free" function yourself, unlike in C.

When you call buf.toString("utf8"), Node walks the byte sequence and decodes it according to the UTF-8 algorithm, producing a new JavaScript string stored internally as UTF-16. This is a real decoding step, not a reinterpretation of memory, which is why invalid byte sequences for the chosen encoding can produce the Unicode replacement character (\uFFFD) instead of throwing.

Common Mistakes

Mistake 1: Using the deprecated new Buffer() constructor

// Deprecated and unsafe - avoid this
const buf = new Buffer(10);
console.log(buf); // may contain leftover memory data

new Buffer(size) has been deprecated since Node 6 because passing it a number silently behaved like the unsafe allocator, and passing it user input opened a security hole (attackers could read stale memory). Use the explicit factory methods instead:

const buf = Buffer.alloc(10);
console.log(buf); // always zero-filled, safe by default

Mistake 2: Sending an allocUnsafe buffer before fully overwriting it

const buf = Buffer.allocUnsafe(16);
buf.write("hi"); // only writes 2 bytes; the other 14 are uninitialized memory
socket.write(buf); // leaks whatever bytes happened to be in that memory region

Because allocUnsafe skips zero-filling for speed, any bytes you don’t explicitly write keep their old contents — which could be fragments of a previous request, a password, or anything else that memory held before. Only use allocUnsafe when you are about to fill every byte yourself; otherwise use the safe allocator:

const buf = Buffer.alloc(16);
buf.write("hi");
socket.write(buf); // remaining bytes are guaranteed to be zero

Mistake 3: Concatenating buffers with the + operator

const a = Buffer.from("foo");
const b = Buffer.from("bar");
const combined = a + b; // coerces both to strings first
console.log(combined); // "foobar" - a string, not a Buffer

The + operator forces both buffers through their default string coercion, discarding the actual binary data and any non-UTF-8 bytes. Use Buffer.concat() to combine buffers and stay in binary form:

const a = Buffer.from("foo");
const b = Buffer.from("bar");
const combined = Buffer.concat([a, b]);
console.log(combined.toString()); // "foobar", still a real Buffer underneath

Best Practices

  • Default to Buffer.alloc(); reach for Buffer.allocUnsafe() only in hot paths where you immediately overwrite every byte.
  • Always pass an explicit encoding to toString() and Buffer.from() rather than relying on the utf8 default, especially when the data source is uncertain.
  • Use Buffer.byteLength(), never string.length, when sizing a buffer for a string that may contain non-ASCII characters.
  • Compare buffer contents with Buffer.compare() or buf.equals(), not ===, which only checks object identity.
  • For large files, prefer streaming (fs.createReadStream) over reading the whole file into one buffer with fs.readFile, to avoid holding megabytes of data in memory at once.
  • Use buf.subarray() instead of the older buf.slice() naming when writing new code; both return a view over the same memory, not a copy.
  • Never log a buffer that might contain secrets (tokens, passwords) — it will print as recoverable hex or base64.

Practice Exercises

  1. Write a function byteStats(str) that returns an object { chars: number, bytes: number } comparing str.length to Buffer.byteLength(str). Test it with a string containing emoji.
  2. Write a script that reads a small file with fs.promises.readFile, checks its first four bytes against the ZIP file signature (0x50, 0x4B, 0x03, 0x04), and logs whether it’s a ZIP archive.
  3. Implement a function xorBuffer(buf, key) that XORs every byte of buf with a single-byte key and returns a new buffer. Verify that running the result through the same function with the same key restores the original bytes. (Note: this is for learning byte manipulation only — single-byte XOR is not real encryption.)

Summary

  • Buffer is Node’s built-in class for raw, fixed-length binary data, available globally and built on top of Uint8Array.
  • Buffer.alloc() is safe and zero-filled; Buffer.allocUnsafe() is faster but may expose old memory until fully overwritten.
  • Encodings (utf8, hex, base64, etc.) only apply when converting between buffers and strings — the underlying bytes have no encoding by themselves.
  • String length and byte length differ for multi-byte characters; use Buffer.byteLength() to get the accurate count.
  • Combine buffers with Buffer.concat() and compare them with Buffer.compare() or .equals() — never use + or ===.
  • Avoid the deprecated new Buffer() constructor; use Buffer.from(), Buffer.alloc(), or Buffer.allocUnsafe() instead.