Fix MongoServerError: Exceeded memory limit for $group stage — allowDiskUse:true and Beyond

TL;DR

MongoDB cut you off at 100MB RAM per pipeline stage. Add allowDiskUse: true to unblock the query immediately — it spills intermediate data to disk instead of crashing. For anything running in production, pair that with a $match early in the pipeline so less data reaches the heavy stages.

// Quick fix — Node.js driver
db.collection('orders').aggregate(
  [
    { $group: { _id: '$customerId', total: { $sum: '$amount' } } },
    { $sort: { total: -1 } }
  ],
  { allowDiskUse: true }  // ← add this
);

Why this happens

Stages like $group, $sort, and $bucket are blocking — they must accumulate all their input before producing any output. MongoDB caps that accumulation at 100MB of RAM per stage. Hit the limit and MongoDB halts the query rather than letting it eat all available memory.

A few situations that reliably trigger this:

• Grouping over a large, unfiltered collection (tens of millions of documents)
• Running $sort before $limit — MongoDB has to sort everything before it can trim
• Building arrays with $push or $addToSet across many documents
• Shared clusters with a low wiredTigerCacheSizeGB (e.g., 0.5GB on a 1GB instance)

Fix 1: allowDiskUse (the escape hatch)

Disk spilling lets MongoDB write a stage's intermediate state to temp files rather than keeping everything in RAM. The query finishes — just slower. On a fast NVMe SSD, expect 5–20x the latency of an in-memory run. On spinning disks or network-attached storage, it's worse.

mongosh / mongo shell

db.orders.aggregate(
  [
    { $group: { _id: '$customerId', total: { $sum: '$amount' } } }
  ],
  { allowDiskUse: true }
);

Node.js (mongodb driver)

const cursor = collection.aggregate(pipeline, { allowDiskUse: true });
const results = await cursor.toArray();

Python (pymongo)

results = list(collection.aggregate(pipeline, allowDiskUse=True))

Mongoose

const results = await Order.aggregate(pipeline).allowDiskUse(true);

Fine for: one-off reports, admin exports, data migrations. Not fine for: APIs serving user requests or dashboards refreshing every few seconds. If this pipeline runs frequently, the next fixes matter more.

Fix 2: Filter early to shrink the working set

Put $match first. That one change usually does more for memory than anything else — MongoDB processes fewer documents through every expensive stage that follows. A collection with 50M orders filtered down to last month's completed ones might drop to 200K documents before hitting $group.

db.orders.aggregate([
  // ✅ Filter BEFORE grouping — uses indexes, shrinks working set
  { $match: {
    createdAt: { $gte: new Date('2024-01-01') },
    status: 'completed'
  }},
  { $group: {
    _id: '$customerId',
    total: { $sum: '$amount' },
    count: { $sum: 1 }
  }}
]);

Without an index on those $match fields, MongoDB still scans the whole collection. Create one:

db.orders.createIndex({ createdAt: 1, status: 1 });

Fix 3: Project only what you need

Drop fields right after $match. Documents with embedded arrays or large text fields can be kilobytes each. Strip everything the $group doesn't touch and working set size can fall 50–80%.

db.orders.aggregate([
  { $match: { status: 'completed' } },
  // Drop heavy fields (e.g., embedded line items array) early
  { $project: { customerId: 1, amount: 1, _id: 0 } },
  { $group: {
    _id: '$customerId',
    total: { $sum: '$amount' }
  }}
]);

Fix 4: Avoid unbounded $push / $addToSet

A $push inside $group adds one entry per document in that group. Across a million orders split among 10,000 customers, that averages 100 items per array — which compounds fast when each item carries any payload. Count instead, or cap the array size after grouping.

// ❌ Can produce huge arrays
{ $group: { _id: '$userId', items: { $push: '$productId' } } }

// ✅ Count instead, if that's all you need
{ $group: { _id: '$userId', itemCount: { $sum: 1 } } }

// ✅ Or cap the array size after grouping
{ $project: { items: { $slice: ['$items', 100] } } }

Fix 5: Raise the server-level limit (MongoDB 6.0+)

Since MongoDB 6.0, internalQueryMaxBlockingSortMemoryUsageBytes controls the per-stage ceiling. The default is 104,857,600 bytes (exactly 100MB). Raising it gives more headroom — but a runaway query can now consume more RAM before failing, so adjust with that tradeoff in mind.

// Raise limit to 500MB (default is 100MB = 104857600 bytes)
db.adminCommand({
  setParameter: 1,
  internalQueryMaxBlockingSortMemoryUsageBytes: 524288000
});

MongoDB Atlas doesn't expose this parameter. There, your only options are to optimize the pipeline or upgrade the cluster tier.

Verification: confirm the fix worked

Run the pipeline with explain to see what happened at each stage:

const explain = await db.orders.aggregate(
  pipeline,
  { allowDiskUse: true, explain: true }
).toArray();

console.log(JSON.stringify(explain, null, 2));

Look for usedDisk: true in the stage output — that confirms the stage spilled to disk. No usedDisk key (or false) means it ran in memory. In production, compare operationTime before and after your changes to measure the actual improvement.

Quick reference

• Immediate fix: allowDiskUse: true in aggregate options
• Best practice: $match first, $project second, group last
• Index check: ensure $match fields are indexed
• Avoid: unbounded $push/$addToSet in large collections
• Atlas users: scale up tier or optimize pipeline — no server parameter access