Failover in Practice

A MongoDB replica set is designed so that when the primary node disappears — a crash, a network partition, a maintenance restart — one of the secondaries is automatically promoted to take its place. This is called failover. It sounds simple in a diagram, but in practice it takes several seconds, it changes what your application can and cannot do during that window, and it interacts with write concern in ways that can silently lose data if you get it wrong. This lesson walks through what actually happens on the wire and on disk during an election, how to trigger and watch one yourself, and how to write client code that survives it gracefully.

Overview: How Failover Actually Works

A replica set is one primary plus one or more secondaries, all replicating the same stream of writes (the oplog) from the primary. Every member sends heartbeat pings to every other member roughly every two seconds. If a majority of voting members stop receiving heartbeats from the primary for longer than electionTimeoutMillis (10 seconds by default), the replica set considers the primary down and starts an election.

The election protocol is derived from Raft consensus. Any eligible secondary can nominate itself as a candidate, request votes from the other voting members, and become primary once it has votes from a strict majority of the set’s voting members (not just a majority of who happens to be reachable). This is why replica sets are normally sized with an odd number of voting members: with 3 members you can lose 1 and still reach a majority (2 of 3); with 2 members you can lose none, because losing either one leaves the survivor with only 1 of 2 votes — not a majority. A member’s eligibility to become primary also depends on its priority (0 means it can never become primary, useful for a reporting/analytics-only secondary) and how caught up its oplog is relative to the old primary.

From the application’s point of view, failover is not instantaneous. During the detection-plus-election window — typically a few seconds up to the full election timeout — there is briefly no primary at all. Any write, and any read using the default primary read preference, will fail or block until a new primary is elected and the driver’s topology monitor discovers it. Reads issued with a secondary read preference generally keep working throughout, because secondaries are still there answering queries; only their ability to accept writes changes.

There is one more subtlety that matters a lot in practice: if the old primary had accepted a write but that write had not yet replicated to a majority of the set when the primary went down, that write can be rolled back once the old primary rejoins the set as a secondary — even though the client may have already received an acknowledgement for it, if it was acknowledged with a weak write concern. This is the single biggest reason write concern choice and failover behavior are taught together rather than as separate topics.

Syntax

You don’t “call” failover directly — it’s triggered automatically by heartbeat loss, or you can trigger it deliberately for testing. The commands you’ll use to inspect and influence it are:

// General shape of the commands used to inspect and influence failover
rs.status();                                            // current member states
rs.stepDown(stepDownSecs, secondaryCatchUpPeriodSecs);  // force the primary to step down
rs.reconfig(configDocument);                            // change member priorities/votes
db.hello();                                             // ask this node if it is currently primary
Command / Field Purpose
rs.status() Returns the current state of every member: stateStr (PRIMARY, SECONDARY, RECOVERING, ARBITER, DOWN, etc.), health, and how far behind the primary each secondary’s oplog is.
rs.stepDown(stepDownSecs, secondaryCatchUpPeriodSecs) Forces the current primary to give up its role for at least stepDownSecs seconds. secondaryCatchUpPeriodSecs is how long the primary will wait for a lagging secondary to catch up before stepping down anyway; if no eligible secondary catches up in time, the step-down is refused.
rs.reconfig(configDocument) Applies a new replica set configuration — used to change a member’s priority, votes, or arbiterOnly flag, all of which affect who is eligible to win an election.
db.hello() The modern replacement for the old isMaster command. Returns isWritablePrimary: true if the node you’re connected to is currently the primary.

Examples

Example 1: Inspecting current replica set state

rs.status().members.map(m => ({
  name: m.name,
  state: m.stateStr,
  health: m.health,
  uptime: m.uptime
}));

[
  { name: 'mongo1:27017', state: 'PRIMARY', health: 1, uptime: 48213 },
  { name: 'mongo2:27017', state: 'SECONDARY', health: 1, uptime: 48210 },
  { name: 'mongo3:27017', state: 'SECONDARY', health: 1, uptime: 48198 }
]

This is the first thing to run whenever failover behavior is in question. health: 1 means the member is reachable via heartbeats; a member that’s down will show health: 0 and a stateStr of DOWN or UNKNOWN from the perspective of members that can’t reach it.

Example 2: Forcing a failover on purpose

You can, and should, test failover deliberately rather than only discovering how your app behaves during a real outage. Connected to the current primary:

rs.stepDown(60, 30);

MongoNetworkError: connection to mongo1:27017 closed

rs.stepDown() makes the primary step down for at least 60 seconds and, in doing so, closes existing client connections to force clients to reconnect and rediscover the new primary — which is exactly why your own shell connection drops. Reconnect to any member and check who won the election:

db.hello().isWritablePrimary;

true

Within a few seconds a different node (mongo2 or mongo3) reports isWritablePrimary: true, and rs.status() will show it as the new PRIMARY while mongo1 sits as SECONDARY for the 60 seconds you specified.

Example 3: A driver that survives failover instead of crashing on it

import { MongoClient } from "mongodb";

const uri =
  "mongodb://:@mongo1:27017,mongo2:27017,mongo3:27017/mydb" +
  "?replicaSet=rs0&retryWrites=true&w=majority";

const client = new MongoClient(uri, { serverSelectionTimeoutMS: 5000 });

try {
  await client.connect();
  const orders = client.db("mydb").collection("orders");

  const result = await orders.insertOne({ item: "widget", qty: 4, status: "pending" });
  console.log("inserted:", result.insertedId);
} catch (err) {
  if (err.name === "MongoServerSelectionError") {
    console.error("no primary available within timeout, giving up:", err.message);
  } else {
    throw err;
  }
} finally {
  await client.close();
}

inserted: new ObjectId("64f1c2a1e4b0f5a1d8c9e123")

Listing all three hosts (instead of just one) lets the driver discover the whole topology and track who the current primary is. retryWrites=true (the modern driver’s default) means a single retryable write error caused by an in-progress election is retried once automatically, transparently, instead of surfacing to your code. serverSelectionTimeoutMS: 5000 caps how long a write will block waiting for a primary to exist before giving up with a clear MongoServerSelectionError — far better than hanging indefinitely or throwing a cryptic network error.

How It Works Step by Step

Walking through an unplanned failover from the moment the primary disappears:

  1. Heartbeat loss. Secondaries and the primary exchange heartbeats roughly every 2 seconds. When a secondary stops hearing from the primary, it starts a timer.
  2. Election timeout. If the primary stays unreachable for electionTimeoutMillis (10 seconds by default), an eligible secondary calls for an election.
  3. Voting. The candidate requests votes from the other voting members. Each voter checks that the candidate’s oplog is at least as up to date as its own before voting yes. A member votes for at most one candidate per term.
  4. Majority reached. Once the candidate collects votes from a strict majority of the set’s voting members, it becomes the new primary and starts accepting writes.
  5. Topology discovery. Drivers connected to the set are continuously monitoring all members in the background (SDAM — Server Discovery and Monitoring). Once the driver observes the new primary via its own hello/heartbeat cycle, it routes new writes there.
  6. Old primary rejoins as secondary. When the original primary comes back (network heals, process restarts), it steps down permanently, catches up its oplog from the new primary, and rejoins as a secondary. Any writes it had accepted but not yet replicated to a majority before it went down are rolled back to keep all members consistent.

Notice that steps 1–4 alone can easily take 10–15 seconds. That’s the realistic window during which your application cannot write (and cannot do primary-preference reads) — plan your retry and timeout settings around that, not around “failover is instant.”

Common Mistakes

Mistake 1: Acknowledging writes with w: 1 and assuming they’re safe

db.orders.insertOne(
  { item: "widget", qty: 4, status: "pending" },
  { writeConcern: { w: 1 } }
);

w: 1 only requires the primary itself to acknowledge the write — it says nothing about whether any secondary has it yet. If that primary crashes before replicating the write, the write is rolled back once a new primary takes over, even though your application already received a success response for it. For anything that matters, require the write to reach a majority of the set:

db.orders.insertOne(
  { item: "widget", qty: 4, status: "pending" },
  { writeConcern: { w: "majority" } }
);

Mistake 2: Connecting to one host with no failover awareness

import { MongoClient } from "mongodb";

const client = new MongoClient("mongodb://:@mongo1:27017/mydb");
await client.connect();
const orders = client.db("mydb").collection("orders");
await orders.insertOne({ item: "widget", qty: 4 });

Pointing the driver at a single host with no replicaSet parameter means it never learns about the other members, can’t discover a new primary after a failover, and has no retry or timeout policy configured. It also has no readPreference or write concern set explicitly, so behavior falls back to defaults you didn’t choose deliberately. List every member, name the replica set, and set explicit timeouts:

const uri =
  "mongodb://:@mongo1:27017,mongo2:27017,mongo3:27017/mydb" +
  "?replicaSet=rs0&retryWrites=true&w=majority";
const client = new MongoClient(uri, { serverSelectionTimeoutMS: 5000 });

Mistake 3: Running a 2-member replica set and expecting real failover

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "mongo1:27017" },
    { _id: 1, host: "mongo2:27017" }
  ]
});

With exactly two voting members, a strict majority means both votes. If either node goes down, the survivor has only 1 of 2 votes — not a majority — so it cannot elect itself primary, and the entire set becomes read-only until the missing member returns. A two-member replica set gives you replication, but not automatic failover. Add a third voting member, either a full data-bearing node or a lightweight arbiter, so the set keeps an odd number of voters:

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "mongo1:27017" },
    { _id: 1, host: "mongo2:27017" },
    { _id: 2, host: "mongo3:27017", arbiterOnly: true }
  ]
});

Best Practices

  • Use an odd number of voting members (3, 5, 7…) so the set can always compute a clear majority after losing one node.
  • Use writeConcern: { w: "majority" } for any write you can’t afford to lose to a rollback; reserve w: 1 for genuinely disposable data like ephemeral logs.
  • List every member host in the connection string (or use a mongodb+srv:// record) and include replicaSet=<name> so the driver can discover the full topology.
  • Set an explicit serverSelectionTimeoutMS so writes fail fast and predictably during an election instead of hanging indefinitely.
  • Leave retryWrites and retryReads enabled (the modern driver’s default) so single, brief errors caused by an in-progress election are absorbed transparently.
  • Give any secondary you don’t want to become primary (a reporting replica, a delayed backup member) priority: 0 explicitly, rather than hoping it never wins an election.
  • Test failover on purpose with rs.stepDown() in staging, under realistic application load, before you’re forced to learn its behavior during a real incident.
  • Monitor rs.status() (or your ops tooling’s equivalent) for member health and replication lag, not just for whether the primary is currently reachable.

Practice Exercises

  • On a local 3-node replica set, run rs.status() and identify the stateStr of each member. Then run rs.stepDown(60, 30) on the primary and immediately re-run rs.status() from a different connection — note which member becomes PRIMARY and how the old primary’s state changes.
  • Write a small mongosh loop that calls db.hello().isWritablePrimary once per second right after issuing a step-down, and record how many seconds pass before a new primary is reachable.
  • Using the Node.js driver, write a script that inserts documents in a tight loop with retryWrites: true while you manually step down the primary in another terminal. Count how many inserts, if any, throw an error versus how many are silently retried.

Summary

  • Failover is an automatic election among secondaries, triggered when a majority of voting members can’t reach the primary for longer than the election timeout (10 seconds by default).
  • During detection and election there is briefly no primary at all — writes and primary-preference reads fail or block until a new primary is discovered.
  • A candidate needs votes from a strict majority of voting members to win, which is why odd-sized voting sets matter and 2-member sets don’t provide real failover.
  • Writes acknowledged with weak write concern (w: 1) can be rolled back after a failover if they hadn’t reached a majority; use w: "majority" for anything important.
  • rs.stepDown() lets you trigger and rehearse failover deliberately instead of only experiencing it during a real outage.
  • Resilient client code lists all replica set members, sets a sane serverSelectionTimeoutMS, and relies on retryWrites/retryReads rather than hand-rolled retry loops.