Fix 'EMFILE: too many open files' Error in Node.js When Processing Files Concurrently

The Error

Error: EMFILE: too many open files, open '/var/app/uploads/image.png'
    at Object.openSync (node:fs:596:3)
    at Object.readFileSync (node:fs:464:35)

Your process dies mid-batch. No warning, no graceful shutdown — just a crash, usually while reading uploads, resizing images, or spawning child processes in parallel.

Root Cause

Every OS caps how many file descriptors one process can hold open at the same time. Linux defaults to 1024. macOS is stricter at 256.

Node.js is non-blocking by design. Map over an array of 5000 files and call fs.readFile on each with no throttle, and Node fires all 5000 operations nearly simultaneously. The OS rejects the 1025th open call with EMFILE.

Two separate problems can cause this:

• The OS limit is too low for your workload
• Your code opens files faster than it closes them — no concurrency control

Fix 1: Increase the File Descriptor Limit (ulimit)

Start by checking what you're currently working with:

ulimit -n        # soft limit
ulimit -Hn       # hard limit

Raise the soft limit for the current shell session:

ulimit -n 65536

To make it survive reboots, edit /etc/security/limits.conf on Linux:

# /etc/security/limits.conf
*    soft    nofile    65536
*    hard    nofile    65536

Running a systemd service? Add this to the [Service] section of your unit file:

[Service]
LimitNOFILE=65536

Reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart your-app

On macOS, the persistent equivalent is:

sudo launchctl limit maxfiles 65536 200000

Verify the process picked up the new limit

# Get PID of your Node process
pgrep -f 'node'

# Check its actual fd limit
cat /proc/<PID>/limits | grep 'open files'

Fix 2: Use graceful-fs (Drop-in Replacement)

graceful-fs patches Node's built-in fs module to queue operations when EMFILE is hit, rather than crashing. Zero config required.

npm install graceful-fs

Swap the import and nothing else changes:

// Before
const fs = require('fs');

// After
const fs = require('graceful-fs');

ES module syntax:

import { readFile, writeFile } from 'graceful-fs';

Under the hood, graceful-fs retries EMFILE errors with exponential backoff. Webpack, npm, and Gulp already use it for exactly this reason — it's battle-tested on large file pipelines.

Fix 3: Limit Concurrency in Your Code

Raising limits and patching fs treat symptoms. The root fix is controlling how many files you open at once.

Bad pattern — opens all files simultaneously

// With 5000 paths, this fires 5000 readFile calls at once
const contents = await Promise.all(
  filePaths.map(p => fs.promises.readFile(p))
);

Option A: Use p-limit to cap concurrency

npm install p-limit

import pLimit from 'p-limit';
import { readFile } from 'fs/promises';

const limit = pLimit(20); // max 20 open files at once

const contents = await Promise.all(
  filePaths.map(p => limit(() => readFile(p)))
);

Twenty concurrent reads is usually a safe ceiling on most Linux boxes without tuning. Adjust based on your workload and the median file size.

Option B: Process files in sequential batches

import { readFile } from 'fs/promises';

async function readInBatches(paths, batchSize = 50) {
  const results = [];
  for (let i = 0; i < paths.length; i += batchSize) {
    const batch = paths.slice(i, i + batchSize);
    const batchResults = await Promise.all(batch.map(p => readFile(p)));
    results.push(...batchResults);
  }
  return results;
}

Option C: Stream large files instead of loading them into memory

import { createReadStream, createWriteStream } from 'fs';
import { pipeline } from 'stream/promises';

async function processFile(inputPath, outputPath) {
  const source = createReadStream(inputPath);
  const dest = createWriteStream(outputPath);
  await pipeline(source, dest);
  // fd released automatically when stream ends
}

Streams release file descriptors as soon as they finish. For large file pipelines — think video processing or bulk image conversion — this is almost always the right model.

Fix 4: Hunt Down Leaked File Descriptors

A steadily climbing fd count points to a leak: files opened but never closed. Check your running process:

# Count open fds
ls /proc/<PID>/fd | wc -l

# See exactly what's open
ls -la /proc/<PID>/fd

If that number grows over time and never drops, you have a leak. Common culprits:

• Calling fs.open() without a matching fs.close()
• Unhandled promise rejections that skip the cleanup path
• Child processes spawned but never awaited

Always close in a finally block so errors can't skip cleanup:

const fd = fs.openSync(path, 'r');
try {
  // ... work with fd
} finally {
  fs.closeSync(fd);
}

Recommended Approach

For production Node.js apps that process files in bulk, stack all three layers — each one catches what the others miss:

• Set LimitNOFILE=65536 in your systemd unit (one-time config, survives deploys)
• Install graceful-fs as a safety net for spikes
• Use p-limit or batch processing to keep concurrency predictable

Small workloads might recover with just one of these. Under real production load — think 10,000+ files per run — you want all three in place before the incident happens, not after.

Verification

Once the fixes are in, watch the fd count while your workload runs:

# Live fd count, refreshed every second
watch -n 1 'ls /proc/$(pgrep -f node)/fd | wc -l'

A healthy process holds steady — the number fluctuates but doesn't climb indefinitely. If it plateaus and your app stops crashing, you're done.