Fix OSError: [Errno 24] Too many open files in Python

TL;DR — Quick Fix

Your process opened more files (or sockets) than the OS allows and never closed them. You've hit the file descriptor limit.

Fastest fix in production: raise the limit for the running user:

ulimit -n 65536

That buys you breathing room. Then fix the root cause before the next deploy — ulimit alone won't save you forever.

What This Error Looks Like

Traceback (most recent call last):
  File "worker.py", line 47, in process
    f = open(path, 'r')
OSError: [Errno 24] Too many open files

Sockets fail the same way:

OSError: [Errno 24] Too many open files
  File "server.py", line 112, in accept
    conn, addr = self.sock.accept()

Same root cause either way. File descriptors (FDs) are shared across files, sockets, pipes, and some IPC mechanisms. You've simply run out.

Root Cause

Every open file or socket consumes one file descriptor. Linux and macOS enforce a per-process soft limit — typically 1024 on Linux. Check yours:

ulimit -n
# or
cat /proc/sys/fs/file-max  # system-wide hard cap

Two culprits account for nearly every case:

FD leak: code opens files or sockets but never closes them. They pile up silently until you hit the wall.
Legitimate exhaustion: a crawler, log processor, or connection pool genuinely needs more FDs than the default allows.

Find out how many FDs your process has open right now:

# replace PID with your process ID
ls /proc/PID/fd | wc -l

# or with lsof
lsof -p PID | wc -l

Climbing toward the limit and never dropping? That's a leak.

Fix 1 — Stop the Leak (Most Common Fix)

Use with statements. Python's context manager calls close() even when an exception fires mid-block — no manual cleanup needed.

Broken pattern:

f = open('data.csv', 'r')
content = f.read()
# forgot f.close() — if an exception fires above, the FD leaks

Fixed:

with open('data.csv', 'r') as f:
    content = f.read()
# FD released here, no matter what

Sockets work the same way:

import socket

with socket.create_connection(('example.com', 80)) as sock:
    sock.sendall(b'GET / HTTP/1.0\r\n\r\n')
    response = sock.recv(4096)
# socket closed automatically

Fix 2 — Close Explicitly in Long-Running Code

Sometimes you can't use with — think object attributes or class-level handles. Wrap with try/finally instead:

f = open('output.log', 'a')
try:
    f.write(line)
finally:
    f.close()

HTTP sessions have the same trap. Use them as context managers:

import requests

with requests.Session() as session:
    resp = session.get('https://api.example.com/data')
    print(resp.json())

Fix 3 — Raise the OS Limit

High-concurrency servers and bulk file processors can legitimately need thousands of FDs. The default of 1024 is just too low.

Temporary (current shell session only):

ulimit -n 65536

Permanent for a user — edit /etc/security/limits.conf:

# /etc/security/limits.conf
www-data soft nofile 65536
www-data hard nofile 65536

Log out and back in for the change to take effect. Confirm with ulimit -n.

For a systemd service — add to the [Service] section:

[Service]
LimitNOFILE=65536

Then apply it: sudo systemctl daemon-reload && sudo systemctl restart yourservice

Fix 4 — Use the resource Module in Python

Need to raise the limit from inside your script — no systemd, no shell access? The resource module handles it at startup:

import resource

soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print(f'Current limits: soft={soft}, hard={hard}')

# Raise soft limit up to the hard limit
resource.setrlimit(resource.RLIMIT_NOFILE, (hard, hard))

One catch: without root, you can only raise the soft limit up to the hard limit. Anything beyond that requires ulimit or limits.conf.

Fix 5 — Cap Connection Pool Size (requests, aiohttp, databases)

Under heavy load, HTTP clients and DB drivers can open connections faster than they're returned to the pool. Pin the pool size explicitly:

import requests
from requests.adapters import HTTPAdapter

session = requests.Session()
adapter = HTTPAdapter(pool_connections=10, pool_maxsize=20)
session.mount('https://', adapter)
session.mount('http://', adapter)

With asyncio, a semaphore keeps concurrent opens in check:

import asyncio

async def process_files(paths):
    sem = asyncio.Semaphore(100)  # cap at 100 concurrent FDs

    async def open_with_limit(path):
        async with sem:
            # file or network operation here
            pass

    await asyncio.gather(*[open_with_limit(p) for p in paths])

Tune that 100 based on your ulimit. A safe rule of thumb: stay below 50% of your soft limit.

Verify the Fix

Watch the FD count in real time while your workload runs:

# Live FD count for PID
watch -n1 'ls /proc/PID/fd | wc -l'

# Check what limit the process actually sees
cat /proc/PID/limits | grep 'open files'

A plateauing count means the leak is gone. Still climbing? Use lsof to find the culprit:

lsof -p PID | sort -k9 | head -40

Repeated entries of the same file path are your leak — that's exactly where to look.

Quick Diagnostic Checklist

Run ulimit -n — is the default still 1024? Raise it.
Check ls /proc/PID/fd | wc -l — how close to the limit are you?
Search your code for open( calls not inside a with block.
Look for unclosed socket, bare requests.get() calls, or DB cursors.
Using threads or async? Make sure parallel opens are bounded — no unbounded gather() or ThreadPoolExecutor with unlimited workers.