Atlas secondary stuck in RECOVERING after oplog rollover

Hello, I'm 小兔大白糖. An Atlas dedicated replica set stayed yellow for more than a week: one member stuck in RECOVERING, the deploy stuck in UPDATING, while app traffic still worked — the real question is usually not “upgrade the tier again,” but whether a secondary has already fallen off the oplog window.
Oplog rollover means the capped oplog overwrote the resume point a lagging secondary needed. Incremental catch-up is then impossible; only an initial sync or node replacement works, and Atlas customers often need Support to clear the stuck change.
Mental model: treat the oplog as a fixed-length ring tape — new writes keep overwriting the oldest segment. The analogy covers “the window shrinks and resume points vanish”; it does not explain Atlas’s deploy state machine, or how slowly a secondary can apply entries.
This is a teaching postmortem for backend engineers running bulk writes on MongoDB Atlas replica sets, and for anyone who needs a crisp line between RECOVERING and what w=majority does not protect. Timestamps, sizes, and lag figures below are teaching-case readings. 中文版:Atlas 从节点 oplog 被卷掉后一直 RECOVERING。
Reading time: ~18 minutes
Contents
- What is the one-line conclusion?
- What was the impact?
- How does the timeline read?
- How was it found?
- How to investigate?
- What is the root cause?
- What does the architecture look like?
- What can you do — and what not?
- What to do after Support restores the set?
- How should bulk writes be paced?
- Detection checklist for next time
- What to remember?
1. What is the one-line conclusion?
Conclusion: once a secondary falls off the oplog window, only a full rebuild helps — a tier upgrade does not replay overwritten history.
A large, sustained upsert workload made the primary append oplog entries faster than one secondary could apply them. Replication lag grew until that secondary’s required history was overwritten. Incremental catch-up became impossible. The node stayed in RECOVERING and needed a full initial sync.
Atlas would not finish a configuration deploy until all three members were healthy, so the cluster remained UPDATING (“0 of 3 servers complete”) for a long time — more than a week in the teaching case. The application kept serving traffic because the primary and the other secondary were healthy. Fault tolerance was reduced to “do not lose another node.”
Raising the tier (M10 → M20) does not repair the member that has already fallen off the oplog. It does lower the chance of the same failure under a future burst, because secondaries can apply faster and the oplog window is usually larger. Pacing bulk writes matters more than write concern alone.
2. What was the impact?
| Area | What happened |
|---|---|
| Availability | Reads and majority writes continued on the healthy primary and one healthy secondary. |
| Fault tolerance | One voting member was unusable. A second failure could lose majority. |
| Cluster changes | Deploys, failovers-as-a-test, and tier changes were blocked or unsafe while state stayed UPDATING. |
| Data on the bad node | In the teaching case, frozen about 15 days behind the primary (syncedTo 2026-10-27 16:30 HKT). Not a client-visible data loss on the primary. |
| Operations | No self-service “restart this member” or “force resync” in Atlas. |
3. How does the timeline read?
Times below mix Atlas alert timestamps (UTC) and the lag reading (HKT). The rows are a teaching-case signal chain.
| When | Signal |
|---|---|
| 2026-10-26 | REPLICATION_OPLOG_WINDOW_RUNNING_OUT (closed after the window recovered briefly). |
| 2026-10-27 16:30 HKT | Unhealthy secondary’s syncedTo stops here. Later readings never move past it. |
| 2026-10-27 | Metric threshold alert, then closed. |
| 2026-11-03 | HOST_DOWN on the replica set for about 9 minutes, then closed. |
| 2026-11-04 | Oplog-window alert again. |
| 2026-11-09 | UI still yellow / RECOVERING. API state UPDATING. No OPEN / TRACKING alerts on the panel. Disk about half full. |
| 2026-11-10 | Another oplog-window alert. MongoDB patch level moved (8.0.36 → 8.0.37) while the cluster stayed UPDATING. |
| 2026-11-11 | Lag check: healthy secondary 0s; unhealthy secondary ~354 hours behind, still RECOVERING. |
4. How was it found?
- Atlas Overview stayed yellow: banner
Deploying your changes failed; we will try again shortly, plus0 of 3 servers complete (waiting for 3 servers to be healthy, configuring MongoDB). - Member list: one
PRIMARYand oneSECONDARYlater showed green; one member stayedRECOVERING. - Early on, all three rows were yellow. That was the cluster-level
UPDATINGstate, not three dead processes. - A common misread is “scale M10 → M20 first.” Treat a tier change as a hypothesis until disk, lag, and alerts line up — not as the repair for a member already outside the window.
5. How to investigate?
5.1 What Atlas would and would not show
- Cluster state from the Admin API:
UPDATING, tier still M10, not paused. - Alert panel: no OPEN or TRACKING items at investigation time. The failure was a stuck workflow, not a live “disk full” or “host down” alarm.
- Closed alerts told the story: oplog window pressure, then a short host-down, then oplog window pressure again.
- UI
...menu hadTest Resilience, pause, and terminate. It had no per-node restart and no resync. Test Resilienceonly steps down the current primary. Atlas also refuses it unless every member is healthy and lag is under 10 seconds. It does not repairRECOVERING.
5.2 Replication evidence
Checked from a client that could see secondary replication info (teaching case):
| Member | Role at read time | syncedTo | Lag |
|---|---|---|---|
| node-a | SECONDARY | 2026-11-11 (current) | 0s |
| node-b | PRIMARY | — | — |
| node-c | RECOVERING | 2026-10-27 16:30 HKT | ~354 hours |
On a healthy member, dbStats showed disk roughly 33 GB used of ~60 GB (~54%). This was not a full disk.
Oplog on the healthy primary still held on the order of 5–6 days of history when sampled. A secondary stuck on 2026-10-27 was already outside that window, so it could not resume by replaying oplog.
Automation on the reachable member still had a newer config version pending than the version it had successfully applied. That matches the UI retry loop: Atlas keeps trying to finish “configure MongoDB,” and each attempt waits for three healthy servers.
5.3 Business workload
The suspected producer is a multi-million-row upsert job (entity-shaped documents, teaching sketch; UpdateOne + upsert, batches sent with bulk_write). That pattern was not proven by a single slow-query log line as “the” command that killed the node. It is the kind of workload that fills an M10 oplog quickly:
- every upsert is an oplog entry, including no-op-looking overwrites
- a tight loop does not wait for the slowest secondary
- M10 disk IOPS and CPU cap how fast a secondary can apply those entries
Other heavy insert/update/delete jobs would have the same effect. “Upsert” is not special; oplog bytes per second vs secondary apply rate is.
5.4 Ruled out
| Idea | Why it does not fit |
|---|---|
| Disk full | ~54% used on the healthy member. |
| All three nodes down | Primary and one secondary served traffic; lag 0 on the healthy secondary. |
| Restart from the Atlas UI | Control does not exist for Atlas (it exists in Cloud Manager). |
Test Resilience | Restarts the primary only, and only when the set is already healthy. |
| Tier upgrade as the immediate fix | The broken member is ~15 days behind the oplog window. A bigger box does not replay overwritten history. |
| Current CPU saturation as proof | No resource alerts still OPEN during the stuck period. Historical oplog-window alerts are the capacity clue, not a live requirement to scale before resync. |
6. What is the root cause?
writes faster than a secondary can apply
→ replication lag grows
→ oplog is a fixed-size ring, so the oldest entries are overwritten
→ the lagging secondary no longer has a common oplog point
→ incremental replication is impossible
→ only initial sync (full rebuild) or a node replacement can catch up
→ that rebuild is slow or stuck
→ member stays RECOVERING
→ Atlas deploy waits for 3 healthy servers
→ cluster stays UPDATING and retries the failed changeOplog rollover means overwrite, not “the file vanished.” New entries consume a capped collection. When the cap is reached, MongoDB removes the oldest entries. The oplog window is how much clock time those remaining entries cover. Fast writes shrink the window from “several days” toward “hours” even if the byte size stays the same.
Once node-c needed an entry older than the current window, no amount of waiting would let it tail the oplog. Atlas’s automatic recovery is an initial sync. If that sync never finishes, the UI stays on RECOVERING and the deploy stays at 0/3.
7. What does the architecture look like?
Normal steady state: clients write to the primary; both secondaries tail the same oplog and stay inside the window.
Broken state after rollover:
8. What can you do — and what not?
| Action | Effect |
|---|---|
| Keep using the cluster for normal app traffic | OK while primary + one secondary stay up. Avoid another member loss. |
| Pause large bulk upserts until 3/3 are healthy | Do this. More fast writes shrink the window and can knock out the remaining secondary. |
Test Resilience | Do not run. Wrong tool, and prerequisites fail. |
| Pause / terminate the cluster | Do not use as a repair. Terminate destroys the deployment. |
| Scale M10 → M20 now | Does not restore overwritten oplog. Optional only after the set is healthy, as future headroom. |
Ask Atlas Support to resync or replace the RECOVERING member and clear the stuck plan | Required. |
Support request (template)
Subject: Stuck UPDATING — one member RECOVERING and outside the oplog window
Cluster: <project name> / <cluster name>
Tier: M10 dedicated, single-region, MongoDB 8.0, 3-node replica set
State: UPDATING for more than a week.
UI:
- "Deploying your changes failed; we will try again shortly."
- "0 of 3 servers complete (waiting for 3 servers to be healthy, configuring MongoDB)."
- Members: PRIMARY healthy, one SECONDARY lag 0, one member RECOVERING.
Replication:
- Healthy secondary syncedTo "now", lag 0s.
- RECOVERING member syncedTo 2026-10-27 16:30 HKT, about 354 hours behind.
- Sampled oplog window on the healthy primary was about 5–6 days, so incremental catch-up is impossible.
- Disk on a healthy member ~33GB / ~60GB. Not disk-full.
- No OPEN alerts at time of writing.
- Closed alerts: REPLICATION_OPLOG_WINDOW_RUNNING_OUT (Oct 26, Nov 4, Nov 10), HOST_DOWN x3 on Nov 3.
Please:
1. Force a clean initial sync or replace the RECOVERING member.
2. Clear the stuck UPDATING / failed deploy so the cluster returns to IDLE with 3 healthy members.
3. Do not treat an M10→M20 upgrade as the fix for this member. A tier change does not replay overwritten oplog.
We will keep large bulk writers paused until the set is healthy.9. What to do after Support restores the set?
- Confirm API state is idle (not
UPDATING) and the UI shows primary + two secondaries, noneRECOVERING. - Confirm both secondaries report lag near 0 and
syncedTois current. - Only then rerun bulk jobs, with the pacing in section 10.
- Decide separately whether M20 or a larger oplog is worth it for the next multi-million-row job. That is prevention, not the repair for a rolled-off member.
Tier upgrade helps later because:
- more CPU, memory, and disk IOPS raise secondary apply rate
- more disk usually means a larger default oplog, so the same lag is less likely to fall off the window
It is not a guarantee. A tight enough loop can still outrun M20. Pacing and lag checks are the actual control.
10. How should bulk writes be paced?
10.1 What w=2 does and does not do
WriteConcern(w=2) means “do not return until two data-bearing members have acknowledged this write” (primary + any one secondary), and wtimeout aborts the wait.
On a 3-node set that is not the same as “wait until every secondary has applied it.”
| Concern | Protects | Does not protect |
|---|---|---|
w=1 | Primary applied the write. Fastest. | Acknowledged data can still exist only on the primary. |
w=2 or w=majority (same bar on 3 voting nodes) | An ack means two members have it. Survives one node loss. | The slower secondary. The fast secondary can satisfy w=2 while the other falls off the oplog. |
wtimeout | The client fails instead of hanging forever. | The batch is then not durable at that concern. It must be retried or recorded as failed. |
So the snippet that sets w=2 and BATCH_SIZE=2000 with no pause is a durability setting, not a replication-lag brake. If one secondary is healthy, bulk_write returns as soon as that secondary acks, and the sick secondary is ignored.
w=majority is the better default for application writes. Use it for the bulk job too, and add time between batches plus a lag gate.
10.2 Paced, idempotent upserts
Requirements:
- stable filter (
entity_idor another unique business key) so a retry does not duplicate rows ordered=Falseso one bad document does not block the rest of the batch- inspect
BulkWriteError; do not treat “no exception” as the only success check if you later split retries - pause between batches
- back off on write-concern timeout instead of immediately sending the next 2,000
- stop the job if lag or oplog window crosses a limit you cannot wait out
BATCH_SIZE=500 and PAUSE_SECONDS=0.5 are starting points for an M10, not universal constants. Raise batch size only after lag stays near zero. Multi-million-row jobs should be checkpointed so a restart resumes from the last successful key.
import time
from pymongo import MongoClient, UpdateOne
from pymongo.errors import AutoReconnect, BulkWriteError, NetworkTimeout
from pymongo.write_concern import WriteConcern
# Start conservative on a small dedicated tier. Increase only while
# secondary lag stays near 0 and the oplog window stays above 24h.
BATCH_SIZE = 500
PAUSE_SECONDS = 0.5
MAX_RETRIES = 5
client = MongoClient(
"mongodb+srv://<user>:<password>@<cluster-host>/?retryWrites=true&w=majority",
serverSelectionTimeoutMS=10000,
)
collection = client["<database>"].get_collection(
"<collection>",
write_concern=WriteConcern(w="majority", wtimeout=10000),
)
def flush(operations: list) -> None:
"""Write one batch. Back off if the set cannot acknowledge in time."""
delay = PAUSE_SECONDS
for attempt in range(1, MAX_RETRIES + 1):
try:
collection.bulk_write(operations, ordered=False)
time.sleep(PAUSE_SECONDS)
return
except BulkWriteError as exc:
# Write-concern timeout: the batch may be on the primary already.
# Upsert on a stable key makes a retry safe. Other write errors
# (validation, etc.) should be inspected in exc.details and not
# blindly retried forever.
write_concern_error = (exc.details or {}).get("writeConcernError")
if write_concern_error and attempt < MAX_RETRIES:
time.sleep(delay)
delay = min(delay * 2, 30)
continue
raise
except (AutoReconnect, NetworkTimeout):
if attempt == MAX_RETRIES:
raise
time.sleep(delay)
delay = min(delay * 2, 30)
def iter_documents():
"""Yield source rows. Checkpoint the caller so a restart can resume."""
raise NotImplementedError
pending = []
for item in iter_documents():
pending.append(
UpdateOne(
{"entity_id": item["entity_id"]},
{"$set": item},
upsert=True,
)
)
if len(pending) >= BATCH_SIZE:
flush(pending)
pending = []
if pending:
flush(pending)10.3 Lag gate
The application user often cannot run replSetGetStatus. Two practical options:
- Operator check (Atlas → Metrics → Replication Lag and Oplog Window, or a monitoring user):
// on the primary, with a role that can read repl set status
db.adminCommand({ replSetGetStatus: 1 })
// compare each secondary member.optimeDate to the primary
// also: rs.printSecondaryReplicationInfo()- Job rule: before each batch (or every N batches), if max secondary lag is above 30 seconds, or the oplog window is under 24 hours, sleep until it recovers or abort. Do not start a multi-million-row job when any member is
RECOVERING.
Suggested budgets for this tier (tune with real lag graphs):
| Signal | Continue | Slow down | Stop |
|---|---|---|---|
| Max secondary lag | under 5s | 5–30s | over 30s, or any member not SECONDARY/PRIMARY |
| Oplog window | over 48h | 24–48h | under 24h |
| Member state | 1 primary + 2 secondaries | — | any RECOVERING / DOWN |
10.4 Oplog size
On dedicated tiers, oplog size can be increased in cluster configuration. A larger oplog uses disk and lengthens the time a secondary may be offline before it must initial-sync. It does not make a secondary apply faster. Set it only with free disk headroom (in the teaching case the cluster was ~half full, so there was some room, not unlimited room). Confirm the current Atlas control name before changing it; the default is a percentage of disk.
11. Detection checklist for next time
- Overview banner: failed deploy,
0 of N,waiting to be healthy. - Member states: do not treat “all rows yellow” as “all nodes dead” while the cluster is
UPDATING. - Replication lag per secondary, not only the primary operations graph.
- Oplog window vs the lagging member’s
syncedTo. If lag time > window, only initial sync works. - Disk, CPU, and IOPS at the time lag was growing, plus closed
REPLICATION_OPLOG_WINDOW_RUNNING_OUT/HOST_DOWNalerts. - Which job was writing (bulk upsert, backfill, migration), its batch size, and whether it slept.
- Only then choose: wait, resync via Support, or add capacity for the next run.
12. What to remember?
- A green primary does not mean the replica set is healthy. Check every secondary’s lag and the oplog window.
- Falling off the oplog is a one-way door for incremental replication. The fix is initial sync or replacement, which Atlas customers cannot click.
w=majority/w=2protects acknowledged writes. It does not keep the slowest secondary inside the window.- Bulk upserts need a rate limit: batch size, pause, backoff, and a lag/window gate.
- Scaling up reduces how often this happens. It does not repair a member whose resume point is already gone.
- While one member is
RECOVERING, do not start another full-collection upsert. Protect the secondary that is still caught up.