Node.js Streams

What Are Streams?

A stream represents data that arrives (or is sent) in chunks over time, rather than all at once. Instead of calling fs.readFile() and waiting for an entire file to load into memory, you can open a readable stream and process each chunk as it becomes available, keeping memory usage low regardless of file size.

Node.js has four fundamental stream types: Readable (source of data, like a file being read), Writable (destination for data, like a file being written), Duplex (both readable and writable, like a TCP socket), and Transform (a duplex stream that modifies data as it passes through, like gzip compression).

import fs from 'node:fs';

const readStream = fs.createReadStream('./huge-log.txt', 'utf8');
const writeStream = fs.createWriteStream('./huge-log-copy.txt');

readStream.pipe(writeStream);

readStream.on('end', () => {
  console.log('Copy complete!');
});

.pipe() connects a readable stream directly to a writable one, automatically handling chunking and backpressure, without loading the whole file into memory.

The Four Stream Types

Readable   - source of data (fs.createReadStream, http response body)
Writable   - destination for data (fs.createWriteStream, http request body)
Duplex     - both readable and writable (net.Socket)
Transform  - duplex that transforms data in transit (zlib.createGzip)

• Readable streams emit 'data' events (or support .read()/async iteration) as chunks arrive.
• Writable streams expose .write() and .end() to accept chunks.
• .pipe(destination) connects a Readable directly to a Writable, handling flow control automatically.
• Transform streams sit between a source and destination, modifying each chunk as it passes through.

Streams Cheatsheet

Where streams show up throughout Node.js's core APIs.

Why Streams Matter for Memory Efficiency

Loading an entire 2GB file into memory with fs.readFile() requires roughly 2GB of RAM (often more, accounting for encoding overhead) before you can do anything with it. A stream instead processes the file in small chunks, typically 64KB by default, keeping memory usage flat regardless of the total file size.

Chaining Streams Together

Because .pipe() returns the destination stream, you can chain multiple streams together, source, through one or more transforms, to a final destination, composing complex data pipelines from small, reusable pieces.

import fs from 'node:fs';
import zlib from 'node:zlib';

fs.createReadStream('./access.log')
  .pipe(zlib.createGzip())
  .pipe(fs.createWriteStream('./access.log.gz'));

This compresses a log file on the fly, streaming through gzip, without ever holding the full uncompressed or compressed file in memory at once.

Common Mistakes

• Using fs.readFile() for very large files instead of streaming, causing high memory usage.
• Forgetting to handle the 'error' event on streams, letting failures crash the process silently.
• Chaining .pipe() without checking for backpressure-related warnings on slower writable destinations.
• Assuming all streams are files, streams also represent network sockets, HTTP bodies, and compressed data.

Key Takeaways

• Streams process data in chunks over time instead of loading everything into memory at once.
• The four stream types are Readable, Writable, Duplex, and Transform.
• .pipe() connects streams together and manages flow control automatically.
• Streams are essential for large files, network I/O, and real-time data processing.