Backup on both secondaries or a dedicated backup secondary

Question

In my current replica set setup, I have a a primary and 2 secondaries (both a normal secondaries - priority 1 on each, no delay, no hidden). Note that reads and writes are only against the primary.

Backups are occurring at regular intervals using LVM on both secondaries. This seems like overkill to me.

Is it more common practice to have one dedicated backup secondary rather than backup on both secondaries? And set this dedicated backup secondary to have priority 0?

Answer 1

Unless you are actively archiving those LVM snapshots and copying them off the servers, having two live nodes doing snapshotting is not overkill. The alternative would be having a single node that would hold all of your snapshots, meaning you would not be able to restore to a previous state if anything happened to that machine.

One important thing to consider with this setup is that there is always the possibility of the replica set electing one of the secondaries to become a new primary, which means there is a risk of all writes/reads hitting the LVM volume that is being regularly snapshotted. That performance hit might not be severe, but it could impact the clients using the database.

The most common practise I have come across is to dedicate a hidden secondary with priority 0 to do the backups. If your replica set spans multiple geographical locations, it might make sense to have one of those dedicated backup nodes in each location.

By setting priority to 0, you know that the dedicated backup node will never step up and become a primary and by keeping it hidden it will not be advertised to any database clients, so no reads would reach it even if reads from secondaries were allowed.

If you choose to go down that route, make sure to keep the number of votes uneven, for example by making the dedicated backup node have zero votes if you already have three members in the replica set.