Aggregation Pipeline Introduction

The aggregation pipeline is MongoDB’s framework for transforming and summarizing data by pushing documents through a sequence of processing stages. Instead of writing one big query, you build a pipeline: an array of stage objects where each stage takes the documents produced by the previous stage, does something to them, and hands the results downstream. It is MongoDB’s answer to SQL’s GROUP BY, JOIN, and windowing features, all expressed as composable stages. This lesson covers the mental model, the syntax, and the handful of stages you’ll reach for constantly: $match, $group, $sort, $project, and $lookup.

Overview / How the Aggregation Pipeline Works

A pipeline is just an array passed to aggregate(). Each element of the array is a stage document whose single key is the stage operator name (always starting with $), such as { $match: { status: "shipped" } }. MongoDB executes the stages strictly in the order you list them, and the output document stream of stage N becomes the input document stream of stage N+1 — conceptually identical to piping commands together in a Unix shell (cmd1 | cmd2 | cmd3). No stage can “see” documents from before the previous stage ran; each stage only knows the shape of what was handed to it.

This matters for indexes. Only the very first stage or two of a pipeline runs against the actual collection on disk, so only stages at the front (typically $match and $sort) can use an index the way find() would. Once a stage like $group or $project reshapes the documents, everything after it operates purely on in-memory BSON documents that no longer resemble the original collection — there is no index to consult anymore. This is why pipeline stage order is a performance decision, not just a logical one.

$group is the workhorse for aggregation. It collapses many input documents into fewer output documents, keyed by whatever expression you put in _id (a field reference, a computed expression, or null to group everything into one bucket). Alongside _id you list accumulator expressions — $sum, $avg, $min, $max, $push, $addToSet, and others — which MongoDB updates incrementally as it scans each input document for that group. Internally this looks a lot like a SQL GROUP BY with aggregate functions, except the grouping key can be an arbitrary computed expression, not just a column.

$lookup performs a left outer join against another collection in the same database, matching a local field against a foreign field and attaching the matches as an array on each document. Because it queries another collection for every input document (or a batch of them), an unindexed or unbounded $lookup against a huge collection can be one of the most expensive things in a pipeline — more on that in Common Mistakes.

One more internal detail worth knowing: by default, stages like $group and $sort must fit their working set in 100MB of memory. If you’re aggregating over a large enough dataset, you’ll hit an error unless you pass { allowDiskUse: true } as the second argument to aggregate(), which lets MongoDB spill intermediate results to temporary files on disk (at the cost of speed). Finally, aggregate() returns a cursor, exactly like find() does — mongosh happens to auto-iterate and print the first 20 results when you don’t store it in a variable, but in driver code you still need to iterate the cursor or call toArray().

Syntax

The general shape of an aggregation call is:

db.orders.aggregate(
  [
    { $match: { field: value } },
    { $group: { _id: "$field", total: { $sum: "$field" } } }
  ],
  { allowDiskUse: false }
);

Stage	Purpose
`$match`	Filters documents, like a `find()` query filter. Fastest and most useful at the front of a pipeline.
`$group`	Collapses documents into groups keyed by an `_id` expression, computing accumulators per group.
`$sort`	Orders the document stream by one or more fields.
`$project`	Reshapes documents: include, exclude, rename, or compute new fields.
`$limit` / `$skip`	Truncates or offsets the stream, usually for pagination or top-N results.
`$unwind`	Turns each element of an array field into its own separate document.
`$lookup`	Left outer join against another collection in the same database.

The first argument to aggregate() is always the pipeline array. The optional second argument is an options object; the most common option is allowDiskUse, which permits disk spilling for stages that exceed the 100MB in-memory limit.

Examples

Assume a db.orders collection where each document looks roughly like { customer: "Alice Chen", status: "shipped", total: 129.99, items: [ ... ], createdAt: ISODate(...) }.

use ecommerce

Example 1: Counting orders by status

db.orders.aggregate([
  { $group: { _id: "$status", count: { $sum: 1 } } }
]);

Output:

[
  { _id: 'shipped', count: 842 },
  { _id: 'pending', count: 156 },
  { _id: 'cancelled', count: 37 }
]

Here $group uses $status as the grouping key and $sum: 1 as the accumulator, which simply adds one for every document that lands in each group — the standard idiom for counting. Notice the result documents no longer have a status field at all; the grouping key becomes the new _id.

Example 2: Top spenders among shipped orders

db.orders.aggregate([
  { $match: { status: "shipped" } },
  { $group: { _id: "$customer", totalSpent: { $sum: "$total" } } },
  { $sort: { totalSpent: -1 } },
  { $limit: 3 }
]);

Output:

[
  { _id: 'Alice Chen', totalSpent: 4287.5 },
  { _id: 'Marcus Lee', totalSpent: 3912.25 },
  { _id: 'Priya Nair', totalSpent: 3560.75 }
]

This chains four stages: filter to shipped orders only, sum total per customer, sort by spend descending, then keep the top three. Because $match comes first, it can use an index on status to avoid scanning cancelled and pending orders at all before the expensive grouping work even begins.

Example 3: Joining in customer details with $lookup

db.orders.aggregate([
  { $match: { status: "shipped" } },
  {
    $lookup: {
      from: "customers",
      localField: "customerId",
      foreignField: "_id",
      as: "customerInfo"
    }
  },
  { $unwind: "$customerInfo" },
  {
    $project: {
      _id: 0,
      orderTotal: "$total",
      customerName: "$customerInfo.name",
      customerEmail: "$customerInfo.email"
    }
  }
]);

Output:

[
  { orderTotal: 129.99, customerName: 'Alice Chen', customerEmail: 'alice.chen@example.com' },
  { orderTotal: 89.5, customerName: 'Marcus Lee', customerEmail: 'marcus.lee@example.com' }
]

$lookup matches each order’s customerId against _id in the customers collection and attaches an array called customerInfo (an array because, in general, a foreign field could match more than one document). Since customerId is a one-to-one reference here, $unwind flattens that single-element array back into a plain embedded object so $project can pull fields out of it with simple dot paths.

How It Works Step by Step

Walking through Example 2 internally: (1) $match runs first and, if status is indexed, MongoDB performs an index scan (IXSCAN) to fetch only shipped-order documents rather than scanning the whole collection. (2) $group then streams those documents one at a time, computing a hash-table-like structure in memory keyed by customer, adding each document’s total into a running sum for that key. (3) Once every input document has been consumed, $group emits one output document per distinct key. (4) $sort orders those (now much smaller) group documents by totalSpent. (5) $limit discards everything past the third document. You can confirm the index usage yourself by appending .explain("executionStats") to any pipeline:

db.orders.aggregate([
  { $match: { status: "shipped" } },
  { $group: { _id: "$customer", totalSpent: { $sum: "$total" } } }
]).explain("executionStats");

Output (abbreviated):

{
  queryPlanner: {
    winningPlan: {
      stage: 'GROUP',
      inputStage: { stage: 'IXSCAN', indexName: 'status_1' }
    }
  },
  executionStats: { nReturned: 3, totalDocsExamined: 842, executionTimeMillis: 4 }
}

If status had no index, inputStage.stage would read COLLSCAN instead, meaning every document in the collection was examined before filtering — a red flag on any collection of meaningful size.

Common Mistakes

Mistake 1: Filtering after $group instead of before it

A common error is writing the filter as if it applies to the original documents, but placing it after a stage that already discarded the field it depends on:

// Wrong: status no longer exists once $group has run
db.orders.aggregate([
  { $group: { _id: "$customer", totalSpent: { $sum: "$total" } } },
  { $match: { status: "shipped" } }
]);

After $group, each document only has _id and totalSpent — status simply does not exist anymore, so the $match silently matches nothing and the pipeline returns an empty array. It also wastes work by grouping every order, including cancelled and pending ones, before attempting to filter. Filter on raw collection fields before you reshape them:

db.orders.aggregate([
  { $match: { status: "shipped" } },
  { $group: { _id: "$customer", totalSpent: { $sum: "$total" } } }
]);

Mistake 2: Mixing inclusion and exclusion in $project

// Wrong: cannot mix 1 (include) and 0 (exclude) except for _id
db.orders.aggregate([
  { $project: { customer: 1, total: 1, status: 0 } }
]);

MongoDB rejects this with an error along the lines of “Cannot do exclusion on field status in inclusion projection”. A $project stage must be either an inclusion projection (list the fields you want, everything else drops) or an exclusion projection (list the fields you don’t want, everything else stays) — the only field allowed to break that rule is _id. Fix it by picking one mode:

db.orders.aggregate([
  { $project: { customer: 1, total: 1 } }
]);

Best Practices

Put $match (and $sort, when possible) as early in the pipeline as you can, so the first stage can use an index instead of scanning the whole collection.
Use $project or $unset early to drop large or unneeded fields, especially big arrays, before they flow through later, more expensive stages.
Check .explain("executionStats") to confirm the leading stage shows IXSCAN rather than COLLSCAN.
Avoid an unbounded $lookup against a huge foreign collection; index the foreignField and filter both sides with $match before joining.
Remember the 100MB per-stage memory limit; pass { allowDiskUse: true } for large $group or $sort operations that exceed it, understanding it trades speed for capacity.
Store long pipelines in a variable (const pipeline = [ ... ]; db.orders.aggregate(pipeline);) for readability and easier testing of individual stages.

Practice Exercises

Write a pipeline on db.orders that returns the average order total per status. Expected shape: an array of documents like { _id: 'shipped', avgTotal: 142.30 }.
Using $group with an _id expression built from { $year: "$createdAt" }, find how many orders each customer placed in a given year.
Extend the $lookup example so each result also includes the number of items in the order (hint: $size on the items array), sorted by orderTotal descending.

Summary

An aggregation pipeline is an array of stages; each stage’s output feeds directly into the next, like piping commands together.
Only leading stages such as $match and $sort can use an index against the raw collection — later stages operate on already-transformed in-memory documents.
$group collapses documents by an _id expression using accumulators like $sum, $avg, and $push.
$lookup performs a left outer join against another collection; pair it with $unwind to flatten one-to-one matches.
aggregate() returns a cursor, not a plain array — mongosh auto-prints it, but driver code must iterate or call toArray().
Use .explain("executionStats") to verify your pipeline is using an index rather than a full collection scan.

Aggregation Pipeline Introduction

Overview / How the Aggregation Pipeline Works

Syntax

Examples

Example 1: Counting orders by status

Example 2: Top spenders among shipped orders

Example 3: Joining in customer details with $lookup

How It Works Step by Step

Common Mistakes

Mistake 1: Filtering after $group instead of before it

Mistake 2: Mixing inclusion and exclusion in $project

Best Practices

Practice Exercises

Summary

Table of Contents