Replica Sets Explained

A MongoDB replica set is a group of mongod servers that all hold the same data, so that if one server goes down the others keep the database available. One member is elected primary and accepts all writes; the rest are secondaries that continuously copy the primary’s changes. This is MongoDB’s built-in answer to hardware failure, planned maintenance, and read scaling — and it’s the foundation almost every production MongoDB deployment is built on, since a single standalone server has no redundancy at all.

Overview: How Replica Sets Work

Every write to the primary is recorded, in order, in a special capped collection called the oplog (operations log). Each secondary opens a long-running connection to the primary (or to another secondary, in chained replication) and continuously pulls new oplog entries, applying them locally in the exact order they happened. This is why replication is described as logical and statement-based rather than disk-block replication: secondaries replay the same logical operations, not raw bytes.

Because secondaries apply the oplog asynchronously, there is always some amount of replication lag — the gap between when data lands on the primary and when it appears on a secondary. Under normal conditions this is milliseconds, but it can grow under heavy write load or a slow network. This matters directly for read behavior: if you read from a secondary, you may see slightly stale data.

A replica set typically has an odd number of voting members (commonly 3, 5, or 7). When the primary becomes unreachable — it crashes, loses network connectivity, or is manually stepped down — the remaining members run an election. MongoDB’s replication protocol is based on ideas similar to the Raft consensus algorithm: a member calls for a vote, and if it receives votes from a majority of the set’s voting members, it becomes the new primary. This is exactly why an odd number of voting members matters — with an even number, a network split can produce a tie where no side has a true majority, and no primary can be elected until the network heals.

Not every member needs to be a full, votable, data-bearing node. You can configure:

  • A priority (0–1000) that biases which members are preferred as primary; a member with priority 0 can never become primary.
  • A hidden member that replicates data but is invisible to client read preference routing — useful for a dedicated backup or analytics node.
  • A delayed member that intentionally lags behind by a configured number of seconds — a safety net against accidental deletes or bad application writes propagating instantly everywhere.
  • An arbiter, which holds no data at all and exists purely to vote in elections, used to achieve an odd number of voters cheaply when you don’t want to pay for a third full data copy.

Write durability is controlled with write concern. The default, { w: 1 }, only requires acknowledgment from the primary — fast, but if the primary crashes before that write replicates, it can be lost during failover. { w: "majority" } waits until a majority of voting members have the write in their oplog before acknowledging it, which is what most applications should use for anything that must survive a failover. Similarly, read preference controls which member(s) a query is allowed to run against: primary (default, always current), primaryPreferred, secondary, secondaryPreferred, and nearest (lowest network latency, primary or secondary).

Syntax

Replica set administration happens through the rs.* helper methods in mongosh, which wrap underlying admin commands:

rs.initiate(configDocument);   // create a new replica set from this node
rs.add(hostAndPort);           // add a new member
rs.addArb(hostAndPort);        // add a vote-only, data-less arbiter
rs.remove(hostAndPort);        // remove a member
rs.reconfig(newConfigDocument);// apply a modified configuration
rs.status();                   // current health/state of every member
rs.conf();                     // the replica set's stored configuration
rs.stepDown(secs);             // force the current primary to step down
rs.isMaster();                 // (legacy) check this node's role; prefer db.hello()
  • configDocument — an object with _id (the replica set’s name, must match every member’s --replSet startup flag) and members, an array of { _id, host, priority, votes, hidden, secondaryDelaySecs, arbiterOnly } objects.
  • hostAndPort — a string like "mongo2.example.net:27017" identifying the node to add or remove.
  • secs (in rs.stepDown) — how many seconds the stepped-down primary refuses to be re-elected, giving another member a chance to take over.

Examples

Example 1: Initializing a three-member replica set

Assume three mongod instances are already running, each started with --replSet myReplSet. Connect mongosh to the first one and run:

rs.initiate({
  _id: "myReplSet",
  members: [
    { _id: 0, host: "mongo1.example.net:27017" },
    { _id: 1, host: "mongo2.example.net:27017" },
    { _id: 2, host: "mongo3.example.net:27017" }
  ]
});
{ ok: 1 }

The node you ran this on becomes the initial primary (it defaults to the highest priority) and immediately begins accepting connections from the other two nodes, which sync their data from it and switch to secondary state.

Example 2: Checking replica set health

rs.status();
{
  set: "myReplSet",
  members: [
    { _id: 0, name: "mongo1.example.net:27017", stateStr: "PRIMARY", health: 1 },
    { _id: 1, name: "mongo2.example.net:27017", stateStr: "SECONDARY", health: 1 },
    { _id: 2, name: "mongo3.example.net:27017", stateStr: "SECONDARY", health: 1 }
  ],
  ok: 1
}

The stateStr field is the first thing to check when diagnosing a cluster — you want exactly one PRIMARY and the rest SECONDARY with health: 1. A member stuck in RECOVERING or STARTUP2 is still catching up on the initial sync and isn’t yet eligible to vote or serve reads.

Example 3: Writing with majority write concern and reading from a secondary

db.orders.insertOne(
  { customer: "Ada Lovelace", total: 129.99, status: "processing" },
  { writeConcern: { w: "majority", wtimeout: 5000 } }
);
{
  acknowledged: true,
  insertedId: ObjectId("66f1a2b3c4d5e6f708192a3b")
}

Because w: "majority" was specified, this call doesn’t return until at least two of the three members have the write durably in their oplog, so the write can survive a primary failover. Analytics-style reads that can tolerate a little staleness can instead be routed off the primary:

db.orders.find({ status: "processing" }).readPref("secondaryPreferred");

This tells the driver to prefer a secondary for this query, falling back to the primary only if no secondary is reachable — useful for offloading reporting load without touching the primary that handles live writes.

How Elections Work Step by Step

  1. Every member sends periodic heartbeats to every other member. If the primary misses heartbeats for the configured electionTimeoutMillis (10 seconds by default), the other members mark it as unreachable.
  2. An eligible secondary (priority > 0, up to date enough) calls for an election and requests votes from the rest of the set.
  3. Each voting member grants its single vote to at most one candidate per election term, generally to whichever eligible candidate has replicated the most recent data.
  4. If a candidate receives votes from a strict majority of all voting members (not just those currently reachable), it transitions to PRIMARY and starts accepting writes.
  5. The old primary, if it’s still running but lost majority connectivity, steps down into SECONDARY state on its own once it notices it can no longer reach a majority — preventing two primaries from accepting writes at once.

This whole process typically completes in a few seconds, but any writes in flight during the gap are rejected until a new primary is confirmed, which is why clients should retry on the driver’s automatic retryable-write logic rather than failing immediately.

Common Mistakes

Mistake 1: connecting to one host instead of the full replica set.

// Wrong: hardcodes one node; the app breaks if that specific node fails over
const client = new MongoClient("mongodb://mongo1.example.net:27017/mydb");

If mongo1 happens to be the node that fails, the driver has no other seed to fall back to. Always give the driver every member and the replica set name so it can discover the current primary automatically:

const client = new MongoClient(
  "mongodb://mongo1.example.net:27017,mongo2.example.net:27017,mongo3.example.net:27017/mydb?replicaSet=myReplSet"
);

Mistake 2: assuming a default write is safely durable. Code that does db.payments.insertOne(doc) with no write concern uses { w: 1 }, acknowledged as soon as the primary itself has the write — if that primary crashes one second later before replicating, the write can vanish even though the client already got a success response. For anything you can’t afford to lose, use { writeConcern: { w: "majority" } } as shown in Example 3.

Mistake 3: configuring an even number of voting members. Two data nodes with no arbiter means a network partition can leave each side with exactly one vote — neither side has a majority, so no primary can be elected and the set becomes read-only until the partition heals. Either run three full data-bearing members, or two plus a lightweight rs.addArb() arbiter to break ties.

Best Practices

  • Always run at least three voting members (or two plus an arbiter) so elections can reach a clean majority.
  • Set writeConcern: { w: "majority" } for writes where losing data on failover is unacceptable, and understand the latency trade-off you’re making.
  • Spread members across separate physical failure domains — different racks, availability zones, or data centers — so a single outage can’t take down a majority at once.
  • Use a hidden or delayed member for backups and reporting rather than hammering the primary or a serving secondary.
  • Monitor rs.status() and the oplog window (how much history the oplog retains before overwriting itself) — a secondary that falls behind that window needs a full resync.
  • Secure internal replica-set traffic with keyfile or x.509 member authentication and TLS; an unauthenticated replica set lets anyone on the network join as a member.
  • Test failover deliberately (e.g. rs.stepDown() in staging) before you rely on it in production, so your application’s retry logic is proven, not assumed.

Practice Exercises

  • Start three local mongod processes on different ports, each with --replSet testSet, and use rs.initiate() to form a replica set. Confirm with rs.status() that one member reaches PRIMARY and the other two reach SECONDARY.
  • With that set running, kill the primary process and watch the remaining two members in rs.status() until one of them transitions to PRIMARY. Note roughly how long the election takes.
  • Insert a document using writeConcern: { w: "majority" }, then write a query against the same collection using .readPref("secondary"). Explain in your own words why the read might occasionally return slightly older data than the most recent write.

Summary

  • A replica set keeps multiple copies of your data in sync using the primary’s oplog, which secondaries continuously replay.
  • Only the primary accepts writes; elections use a majority-vote protocol to pick a new primary when the old one becomes unreachable.
  • Odd voting-member counts (or an arbiter) avoid split-vote deadlocks during network partitions.
  • Write concern controls durability guarantees per write; read preference controls which members a query is allowed to hit.
  • Hidden, delayed, and arbiter members let you tune a replica set’s role beyond simple primary/secondary redundancy.