Understanding the Error
Few things are as frustrating as seeing (error) CLUSTERDOWN when you try to run a simple command. This error means your Redis cluster has stopped accepting writes and reads because it no longer considers itself healthy enough to guarantee data integrity.
Common Causes
Redis usually flips the CLUSTERDOWN switch for two specific reasons:
- Quorum Loss: This happens when a majority of master nodes disappear. For example, if you have a 6-node cluster (3 masters, 3 replicas) and 2 of your masters go offline simultaneously, the remaining master cannot reach a majority. It stops operations to prevent data inconsistency.
- Unassigned Hash Slots: Redis divides data into exactly 16,384 hash slots. If even one slot is not hosted by an active node—because both a master and its replica failed—the entire cluster shuts down by default.
These failures often stem from a network split, a sudden power loss across multiple rack units, or a cloud provider outage affecting an entire availability zone.
Step-by-Step Recovery Guide
Step 1: Audit the Cluster State
Start by connecting to any node that is still pingable. You need to see the cluster's perspective of the failure. Run the following command:
redis-cli -h <node-ip> -p 6379 CLUSTER NODES | grep fail
This filters for nodes marked as fail or fail?. If you see more than half of your masters listed here, you have confirmed a quorum loss.
Step 2: Bring Missing Nodes Back Online
The fastest fix is simply reviving the original processes. If the servers rebooted or the service crashed, restart Redis on the failed machines:
# On the server that went down
sudo systemctl start redis-server
# Or if using a custom config
redis-server /etc/redis/redis.conf
Once the nodes rejoin, they will automatically sync. The CLUSTERDOWN state should disappear within seconds of the nodes reaching a handshake.
Step 3: Repairing the Cluster via CLI
Sometimes a node is gone for good—perhaps a cloud instance was terminated or a disk died. If you can't bring the old node back, you must reassign its slots to the survivors. Run the fix utility from a healthy node:
redis-cli --cluster fix <healthy-node-ip>:6379
The tool will scan the 16,384 slots and identify the orphans. It will ask for permission to rebind them to the remaining masters. Type yes to proceed. This effectively heals the cluster by ignoring the dead nodes.
Step 4: The "Nuclear" Option (Last Resort)
If the cluster metadata is hopelessly corrupted and --cluster fix fails, you might have to reset the nodes. Warning: This wipes cluster configuration and can lead to data loss.
Run this on every reachable node to turn it back into a standalone instance:
redis-cli CLUSTER RESET HARD
After doing this, you will need to run the --cluster create command again as if you were setting up the environment for the first time.
Step 5: Adjusting Availability Settings
If you prefer your cluster to stay online even if some data is missing, you can change the strictness of the slot coverage. This is useful for caching scenarios where availability matters more than 100% data consistency.
redis-cli CONFIG SET cluster-require-full-coverage no
To make this change survive a reboot, add cluster-require-full-coverage no to your redis.conf file.
Verifying the Fix
Don't assume it's fixed just because the error stopped. Run these three checks:
- Check Status:
redis-cli CLUSTER INFO | grep cluster_state(Should sayok). - Check Coverage:
redis-cli --cluster check <ip>:6379(Should sayAll 16384 slots covered). - Test Traffic:
redis-cli SET health_check 1.
Prevention Tips
- The 10000 Port Rule: Ensure your firewall allows traffic on the base port (6379) AND the cluster bus port (16379). Without the bus port, nodes can't "vote," leading to false quorum failures.
- Increase Timeouts: If you see frequent
CLUSTERDOWNblips during high traffic, increasecluster-node-timeoutto 30000ms (30 seconds) in your config to handle transient network lag. - Odd Number of Masters: Always use at least 3 master nodes. This ensures a clear majority is possible during a split-brain event.