The ProblemWhen Nginx starts dropping requests and your error logs are flooded with could not allocate new entry in shm zone "mylimit" while logging request, it means your shared memory (shm) zone is exhausted. This usually happens on high-traffic sites where Nginx's rate-limiting module is tracking more unique IP addresses than the allocated memory can handle.

Basically, Nginx has run out of slots to store the state of incoming requests, and because it cannot track whether a new IP should be limited, it defaults to rejecting the entry or behaving unpredictably.

The Error in Logs```

2024/05/28 10:15:30 [error] 1234#0: *5678 could not allocate new entry in shm zone "mylimit" while logging request, client: 192.168.1.1, server: example.com, request: "GET /api/v1/resource HTTP/1.1", host: "example.com"

## Immediate Solution: Increase Zone SizeThe most direct fix is to increase the size of the shared memory zone in your Nginx configuration. Nginx allocates a fixed amount of memory at startup for each rate-limiting zone.
- Open your Nginx configuration file (usually `/etc/nginx/nginx.conf` or a specific file in `/etc/nginx/conf.d/`).- Locate the `limit_req_zone` directive.- Increase the size parameter (e.g., from `10m` to `50m` or `100m`).### Example ConfigurationBefore:

http { # 10MB can hold about 160,000 states limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s; }

After:

http { # 100MB can hold about 1.6 million states limit_req_zone $binary_remote_addr zone=mylimit:100m rate=10r/s; }

## Why This Happens: The MathNginx uses shared memory to store states for the `$binary_remote_addr` (or whatever key you use). In a 64-bit system:
- One state entry typically consumes **128 bytes**.- A **1MB** zone can store approximately **8,000 to 16,000** entries depending on the architecture and key length.- A **10MB** zone (standard default in many tutorials) handles about **160,000** unique IPs.If you have a traffic spike, a DDoS attack, or simply a very popular site where hundreds of thousands of unique users visit within a short window, the 10MB limit will be hit quickly.
## Optimization: Using Binary Remote AddressIf you are currently using `$remote_addr`, switch to `$binary_remote_addr`. This is a critical optimization for memory usage.
- `$remote_addr`: Stores the IP as a string (e.g., "192.168.1.1"), which takes 7 to 15 bytes per entry.- `$binary_remote_addr`: Stores the IP in binary format (4 bytes for IPv4, 16 bytes for IPv6).Using the binary version significantly reduces the footprint of each entry in the `shm` zone, allowing you to store more states in the same amount of memory.

Always prefer binary_remote_addr for rate limiting

limit_req_zone $binary_remote_addr zone=mylimit:20m rate=5r/s;

## How to Verify the FixAfter updating the configuration, you must test the syntax and reload Nginx:

Check for syntax errors

sudo nginx -t

If successful, reload the service

sudo systemctl reload nginx

To confirm the error is gone, monitor your error log in real-time while your site is under load:

tail -f /var/log/nginx/error.log | grep "shm zone"

If the error no longer appears, the new memory allocation is sufficient. If it reappears quickly, you may be under a distributed attack or your traffic volume requires an even larger zone (e.g., 200m or 500m).
## Common Pitfalls### 1. Aggressive Rate LimitingIf your `rate` is set too low (e.g., `1r/s`), Nginx keeps the state in memory longer to ensure the client stays within the limit. This fills the zone faster. Evaluate if your rate limits are too strict for your legitimate user base.
### 2. Tracking the Wrong VariableIf you use `$http_user_agent` as a key, and you get thousands of unique (or spoofed) user agents, the memory will fill up almost instantly. Stick to `$binary_remote_addr` unless you have a very specific use case.
### 3. Memory Limits on the ServerDon't just set `zone=mylimit:2g` if your server only has 1GB of RAM. Nginx will fail to start if it cannot allocate the requested shared memory from the OS.
## Lessons Learned- **Pre-calculate capacity:** Estimate your peak unique visitors per minute and allocate at least 1MB per 10,000 expected concurrent unique IPs.- **Monitor RAM:** Shared memory zones are allocated at startup. Ensure your system has enough free physical RAM to accommodate large zones.- **Use Binary:** Always use `$binary_remote_addr` to maximize the number of entries per megabyte.- **Cleanup:** Nginx automatically removes old entries when the zone is full, but it can't keep up if the influx of new IPs is faster than the LRU (Least Recently Used) expiration. Larger zones provide the necessary buffer.