Fixing MongoDB 'WiredTiger error: write failed: No space left on device' Due to Full Disk

TL;DR Quick Fix

Encountering the WiredTiger error: write failed: No space left on device means your MongoDB server's disk has run out of space. To get MongoDB back online quickly, you'll need to free up disk space right away. Here's a rapid checklist:

Check Disk Usage: Use df -h to pinpoint the full partition.
Identify Large Files/Folders: Use sudo du -sh /path/to/directory/* to discover what's consuming space. Pay special attention to /var/lib/mongodb and /var/log.
Clear OS Logs: Delete or truncate old system logs, typically found in /var/log.
Clear MongoDB Logs: Rotate or delete old MongoDB log files (e.g., in /var/log/mongodb).
Delete Unnecessary Files: Remove any old backups, temporary files, or test data not critical for immediate operation.
Increase Disk Space: For a lasting solution, expand the disk volume using your cloud provider or VM settings.

Detailed Root Cause

The WiredTiger error: write failed: No space left on device message is quite clear: your system's disk is completely full. This means MongoDB's WiredTiger storage engine can no longer write any data. WiredTiger, which has been the default storage engine for MongoDB since version 3.2, manages how data is stored on disk, including data files, index files, and journal files.

When the disk runs out of space, WiredTiger cannot perform essential operations:

It cannot write new data or update existing documents.
It cannot create or update index entries.
It cannot write to its journal files, which are critical for data durability and recovery.
It cannot perform internal operations that require temporary disk space.

This issue can arise from several factors:

Rapid Data Growth: Your database might have expanded much faster than anticipated.
Large Log Files: MongoDB logs or system logs (like syslog or journald) could have consumed all available space. For instance, a journalctl log file might grow to several gigabytes if not managed.
Unreclaimed Space: Deleting data doesn't always immediately free up space for the operating system. WiredTiger pre-allocates space and reuses it internally. Additionally, fragmentation can make data files appear large, even when they contain significant internal free space.
Snapshots/Backups: Old database backups or volume snapshots stored on the same disk might be consuming significant space.
Temporary Files: Other applications on the same server may have generated large temporary files.

When this error strikes, MongoDB typically stops responding to write operations. While read operations might still work briefly, internal writes like journal updates will eventually fail too. Your immediate goal is to free up enough disk space to get MongoDB back to normal.

Fix Approaches

1. Diagnose Current Disk Usage

Before you do anything else, it's crucial to understand what's actually filling up your disk.


df -h

This command shows disk space usage for all mounted filesystems. Identify the partition that is 100% full (or very close to it).


sudo du -sh /var/lib/mongodb
sudo du -sh /var/log
sudo du -sh /tmp

Use du -sh to summarize disk usage for key directories. The MongoDB data directory (/var/lib/mongodb by default) is usually the prime suspect and can often consume hundreds of gigabytes. Also, check /var/log for system and application logs, which might take up tens of gigabytes, and /tmp for temporary files.

2. Clear Non-MongoDB Data (Temporary Relief)

Clearing space outside of MongoDB is often the fastest way to get your database operational again, especially if the MongoDB data directory isn't the only problem.

Clear System Logs:

sudo journalctl --vacuum-size=500M # Keep only 500MB of journal logs sudo journalctl --vacuum-time=7d # Keep logs only for the last 7 days sudo apt autoremove # For Ubuntu/Debian, remove unused packages sudo yum clean all # For CentOS/RHEL, clear package cache

  
  - 
    **Clear Temporary Files:**

    ```bash

sudo rm -rf /tmp/*

Exercise caution when deleting files from /tmp, as some applications might be actively using them. However, it's generally safe to remove older files that are no longer in use.

Delete Old Backups/Snapshots: If you have backups stored on the same volume, move them to another storage location or delete old ones you no longer need.

3. MongoDB Specific Cleanup

3.1. Rotate/Clear MongoDB Logs

MongoDB logs can become quite large, quickly consuming disk space. You should configure log rotation using tools like logrotate on Linux, or manually rotate and delete older log files.


# Stop MongoDB (if not already stopped due to disk full)
sudo systemctl stop mongod

# Move current log file to a backup (or delete if not needed)
sudo mv /var/log/mongodb/mongod.log /var/log/mongodb/mongod.log.old

# Restart MongoDB
sudo systemctl start mongod

MongoDB will create a new mongod.log file. You can then safely delete mongod.log.old after verifying MongoDB starts correctly.

To rotate logs without restarting MongoDB, connect to the admin database and run:


db.adminCommand({ logRotate: 1 })

This command renames the current log file and creates a new one. You can then manually delete the old rotated log file from the filesystem.

3.2. Compact Collections/Database

WiredTiger intelligently reuses disk space internally. Consequently, simply deleting documents doesn't always immediately shrink the data files at the operating system level. The compact command helps by defragmenting and reclaiming this unused space within data files. You can run it on specific collections or the entire database.

Warning: Be warned: Compacting can be very resource-intensive, straining CPU and I/O, and it might temporarily degrade performance. For large collections, this process can take a considerable amount of time and demands free disk space equal to the collection's size during the operation. It's best to schedule this during maintenance windows or perform it on a secondary node within a replica set.


// Compact a specific collection (recommended for large collections)
db.runCommand({ compact: "myCollection" })

// Compact the entire database (use with extreme caution on large databases)
db.runCommand({ compact: 1 })

After compacting, remember to check disk space again with df -h and du -sh /var/lib/mongodb.

3.3. Delete Unnecessary Data

If you have old, unused collections or even entire databases, deleting them offers a direct and often significant way to free up space. This is a permanent action, so always ensure you have reliable backups if there's any uncertainty about the data.


// Drop a collection
db.myOldCollection.drop()

// Drop an entire database
use myOldDatabase
db.dropDatabase()

3.4. Repair Database (Last Resort)

The --repair option rebuilds your database from scratch, which can reclaim space and fix potential corruption. This is a drastic measure, however. It requires free disk space equivalent to at least the size of your current data directory to complete, and it also entails significant downtime.

Warning: Crucially, for replica sets, it's generally much safer to resync a secondary node from a healthy primary than to run --repair. Running --repair on a primary will inevitably lead to substantial downtime.


# Stop MongoDB
sudo systemctl stop mongod

# Run repair (ensure you have enough free space, at least the size of your data directory)
mongod --dbpath /var/lib/mongodb --repair

# Start MongoDB
sudo systemctl start mongod

4. Increase Disk Space (Permanent Solution)

Ultimately, the most robust and long-term solution involves provisioning more disk space for your MongoDB instance.

Cloud Providers (AWS, Azure, GCP): You can typically resize your EBS volume (AWS), managed disk (Azure), or persistent disk (GCP) without downtime. For example, you might increase an AWS EBS volume from 100GB to 200GB. Afterward, you'll need to extend the filesystem (e.g., resize2fs or xfs_growfs) on the OS.
```
# After increasing disk size in cloud console/API
# For ext4 filesystems:
sudo resize2fs /dev/sda1 # Replace /dev/sda1 with your actual device

# For XFS filesystems:
sudo xfs_growfs /mount/point # Replace /mount/point with your actual mount point (e.g., /)
```

  
  - 
    **Virtual Machines/On-premises:** This might involve adding a new physical disk, extending an existing LVM volume, or migrating data to a larger drive altogether.

  

### 5. Configure WiredTiger Compression (Preventative)
WiredTiger offers robust compression options for both collections and indexes, which can drastically reduce disk space usage. Keep in mind that this is a preventative measure for future data growth, not an immediate fix for an already full disk.

You can set default compression for new collections/indexes or change it for existing ones:

```javascript

// Set default compression for new collections (in mongod.conf)
storage:
  wiredTiger:
    engineConfig:
      defaultCollectionConfig:
        blockCompressor: snappy # or zlib, zstd, none
      defaultIndexConfig:
        blockCompressor: snappy

// Change compression for an existing collection
db.runCommand({
  collMod: "myCollection",
  storageEngine: {
    wiredTiger: {
      configString: "block_compressor=zlib"
    }
  }
})

Applying compression to an existing collection means rewriting all of its data. This can be a time-consuming process and will consume significant I/O resources. Therefore, it's best to consider doing this during a scheduled maintenance window or on a secondary node.

Verification Steps

Once you've applied any of these fixes, it's crucial to verify that disk space has indeed been reclaimed and that MongoDB is functioning correctly.

Check Disk Space:

df -h


Confirm that the disk partition now has sufficient free space, ideally 10-20% free.

  
  - 
    **Check MongoDB Logs:** Review `/var/log/mongodb/mongod.log` for any new errors or indications that MongoDB has successfully restarted and is accepting connections.

  
  - 
    **Perform a Write Operation:** Connect to MongoDB and perform a small write operation to ensure it can write data without error.

    ```javascript

    use admin
    db.testWrites.insertOne({ timestamp: new Date(), message: "Disk space test successful" })

Then, check the log for any WiredTiger error messages, confirming their absence.

Monitor Database Status: Check replica set status (if applicable) and ensure all members are healthy and in the PRIMARY or SECONDARY state.
```
rs.status()
```

  

## Further Reading

  - [MongoDB Manual: WiredTiger Storage Engine](https://www.mongodb.com/docs/manual/core/wiredtiger/)
  - [MongoDB Manual: compact Command](https://www.mongodb.com/docs/manual/reference/command/compact/)
  - [MongoDB Manual: logRotate Command](https://www.mongodb.com/docs/manual/reference/command/logRotate/)
  - [MongoDB Manual: WiredTiger Compression Configuration](https://www.mongodb.com/docs/manual/reference/configuration-options/#storage.wiredTiger.engineConfig.defaultCollectionConfig.blockCompressor)