debugjavascriptMajor
EMFILE: too many open files when processing directories
Viewed 0 times
EMFILEtoo many open filesfile descriptorulimitgraceful-fsconcurrent files
nodejs
Error Messages
Error: EMFILE, too many open filesError: EMFILE: too many open files, openProblem
Processing many files concurrently with fs.readFile or fs.open causes Error: EMFILE, too many open files. The OS has a per-process file descriptor limit (usually 1024 on Linux/macOS).
Solution
Limit concurrent file operations:
// Option 1: Use graceful-fs (drop-in replacement)
const fs = require('graceful-fs');
// Option 2: Limit concurrency manually
async function processFiles(files, limit = 50) {
const results = [];
for (let i = 0; i < files.length; i += limit) {
const batch = files.slice(i, i + limit);
const batchResults = await Promise.all(batch.map(f => fs.promises.readFile(f)));
results.push(...batchResults);
}
return results;
}
// Option 3: Increase OS limit
// macOS: ulimit -n 10240
// Linux: edit /etc/security/limits.conf
// Option 1: Use graceful-fs (drop-in replacement)
const fs = require('graceful-fs');
// Option 2: Limit concurrency manually
async function processFiles(files, limit = 50) {
const results = [];
for (let i = 0; i < files.length; i += limit) {
const batch = files.slice(i, i + limit);
const batchResults = await Promise.all(batch.map(f => fs.promises.readFile(f)));
results.push(...batchResults);
}
return results;
}
// Option 3: Increase OS limit
// macOS: ulimit -n 10240
// Linux: edit /etc/security/limits.conf
Why
Each open file uses a file descriptor. The OS limits these per process to prevent resource exhaustion. Node.js async I/O opens many files simultaneously without waiting for previous ones to close.
Gotchas
- ulimit changes are per-shell session — make them permanent in .bashrc/.zshrc
- graceful-fs queues open calls and retries on EMFILE
- fs.promises API still has the same limit — it's an OS constraint
- macOS default is 256 soft limit, Linux is 1024
Code Snippets
graceful-fs fix
// Use graceful-fs as drop-in replacement
const fs = require('graceful-fs');
// It queues file operations to avoid EMFILEContext
When reading, writing, or watching many files concurrently in Node.js
Revisions (0)
No revisions yet.