The Problem
It is a classic sysadmin headache: your monitoring tool alerts you that a disk is at 100% capacity. You find a massive 80GB log file, delete it with rm, and wait for the alert to clear. But it doesn't. Even though the file is gone, df -h still insists the drive is full, while du -sh shows plenty of available space.
This discrepancy happens because Linux treats files differently than some other operating systems. If a process is still writing to a file when you delete it, the space isn't reclaimed immediately. You end up with "ghost" files that consume space but don't appear in any directory listing.
TL;DR: The Quick Fix
If you need to reclaim space right now without a reboot:
- Find the process holding the deleted file:
sudo lsof +L1 - Restart the service listed in the output.
- If you cannot stop the service, truncate the file descriptor:
: > /proc/PID/fd/FD_NUMBER
Why the Space Stays Locked
In the Linux filesystem, a file is only truly purged when two conditions are met:
- The link count hits zero (no more filenames point to the data).
- No active processes have an open file descriptor pointing to it.
When you run rm, you remove the link, but you don't necessarily close the file. If Nginx, a database, or a custom Java app is still holding that file open, the kernel keeps the data blocks reserved. df looks at the filesystem's total used blocks, so it still sees that 80GB as occupied.
Step-by-Step Troubleshooting
1. Verify the Discrepancy
Start by confirming that the filesystem and the directory tree disagree on usage. Run:
df -h
Then, check the actual size of existing files, excluding virtual filesystems:
sudo du -sh / --exclude=/proc --exclude=/sys --exclude=/dev
If df reports 100GB used but du only finds 20GB, you definitely have open file descriptors hanging onto deleted data.
2. Find the "Ghost" Files
The lsof (List Open Files) utility is your best friend here. Use the +L1 flag to specifically look for files with a link count of less than one (deleted files):
sudo lsof +L1
Focus on the SIZE column to find the heavy hitters. The output usually looks like this:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
nginx 4567 root 3w REG 253,1 80GB 0 99887 /var/log/nginx/access.log (deleted)
In this real-world example, Nginx (PID 4567) is holding an 80GB file open on file descriptor 3.
3. The Standard Fix: Restart the Service
The cleanest approach is to restart the application. This forces it to close all file descriptors and allows the kernel to finally release the disk blocks.
sudo systemctl restart nginx
If the process is a standalone script, you may need to terminate it manually:
sudo kill -15 4567
4. The Pro Fix: Truncate Without Downtime
Sometimes you cannot afford to restart a service, like a production database or a high-traffic web server. In these cases, you can "empty" the file through the /proc filesystem without stopping the process.
Using the PID (4567) and FD (3) from our previous step, run:
sudo bash -c ': > /proc/4567/fd/3'
This command sends a null signal to the specific file descriptor. It instantly shrinks the file on disk to 0 bytes. The process remains active and the file descriptor stays valid, but your disk space returns immediately.
How to Prevent This Recurring
Manual deletion of active logs is the primary cause of this issue. To keep your disks healthy, follow these practices:
- Configure logrotate: Ensure all logs are managed by
logrotate. It uses signals likeSIGHUPto tell apps to close old logs and start new ones safely. - Never 'rm' an active log: If you must clear a log file manually, don't delete the file. Truncate it instead:
/var/log/heavy-app.log
This wipes the content but keeps the file handle intact, preventing the "ghost file" problem entirely.
- **Monitor for discrepancies:** Set up a check that compares `df` and `du` output to catch these leaks before they trigger an emergency.