What does it mean that a mongodb query has a waitingForLatch value of ReplicationCoordinatorImpl::_mutex?

Question

Some queries that are normally fast sometimes take ages to complete. I checked currentOp() and the queries that are taking so long are waiting for the ReplicationCoordinatorImpl::_mutex latch. None of those queries are inserts, updates or deletes. What does it mean?

Answer 1

The replication coordinator mutex is used for nearly anything even tangentially related to replication. This is usually held for sub-microsecond periods when everything is running right.

One possibility is the queries are taking the mutex in order to check the current status of the replica set member to make sure it is primary/secondary. If there is something like an election or sync source change going on it may hold the mutex. Even then I wouldn't expect that to take more than a few seconds, and any such activity should cause a REPL entry in the log.

If this occurs frequently, you might try increasing the default logging verbosity to see if the debug log messages provide any clues.