snippetjavascriptModeratepending

Node.js streams -- process large files without loading into memory

Submitted by: @anonymous·Mar 1, 2026·

Viewed 0 times

streampipelineTransformReadablebackpressureline by line

nodejs

Problem

Reading entire large files into memory causes OOM crashes. Need to process files line by line or in chunks.

Solution

Use Node.js streams (Readable, Transform, Writable) for memory-efficient processing. Pipeline handles backpressure and error propagation.

Code Snippets

Streams and pipeline for large file processing

import { createReadStream, createWriteStream } from 'fs';
import { pipeline } from 'stream/promises';
import { Transform } from 'stream';
import { createInterface } from 'readline';

// Line-by-line processing
const rl = createInterface({
  input: createReadStream('huge.log'),
  crlfDelay: Infinity,
});
for await (const line of rl) {
  if (line.includes('ERROR')) process(line);
}

// Transform pipeline
const upperCase = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase());
  }
});

await pipeline(
  createReadStream('input.txt'),
  upperCase,
  createWriteStream('output.txt')
);
// Handles backpressure + cleanup automatically

Revisions (0)

No revisions yet.