MongoDB Replica Set member cannot start due “84 key/value already in index”

Question

I have a Replica Set with 3 members (1 primary and 2 secondaries), and one of the secondaries was working just fine and thrown the following error

> 2018-03-12T11:03:54.868-0400 E REPL [repl writer worker 13] writer
> worker caught exception: :: caused by :: 84 key/value already in index
> on: { ts: Timestamp 1520867034000|8, h: 287037468373256260, v: 2, op:
> "u", ns: "Sitecore_analytics_PROD.Interactions", o2:
> 
> { _id: BinData(3, EFC3DD1A15B75442986344FDE9CC71EB) } , o: { $set:
> 
> { ContactVisitIndex: 2 } } } 2018-03-12T11:03:54.868-0400 I - [repl
> writer worker 13] Fatal Assertion 16360 2018-03-12T11:03:54.868-0400 I
> - [repl writer worker 13] 
> ***aborting after fassert() failure

After that the MongoDB cannot start anymore, and keep throwing the same error!

How can I determine the root cause of it? Is there a way to prevent? How to solve it?

Answer 1

Do initial sync for that problematic secondary node. Just delete files from dbpath and start mongod. Node will do initial sync automatically. Other and faster way (if you can) is copy snapshot of dbpath of existing node (can be primary or secondary) and start mongod.

The problem here is that there is already entry at unique index and system tries to apply opLog line and crashes to that error.

Answer 2

As i am able to see from your log files you are getting the error like ***aborting after fassert() failure. As young DBA Blog here or MongoDB Jira here, here and here. Without more information its difficult to determine the exact source of the issue.

As @alernerdev describe here I hope you have data backups? Bring down your faulty secondaries, delete the data on one of them, and bring it back up (just one) -- this will start the initial re-sync process. If all is well, at this point you will have 1 primary and 1 secondary. Before you do this, verify that the size of your oplog window is large enough to complete the initial sync process. Once this is all back to normal, do the same for your second Secondary.

For your further ref here and here