MongoDB Replica-Set Disk Cleanup

Question

I am trying to shrink the size of my MongoDB replica-set(the collections are the same size but disk space keeps growing). According to the MongoDB website, I should just run mongod --repair on the master node to compact all collections. The problem would be downtime for the website. So, I have two options(that I know about):

1. Take secondary node off of replica-set and run mongod --repair on it and restart back on replica-set. I tried this and couldn't get past permission errors on 'local' collection.

2. Shut down secondary node and delete all files in the data directory. Restart mongo and let it recover from master. This actually worked for me but my only concern is, what if your journal collection is full and since it's a capped collection, will you only receive the data that is in the journal or will you actually copy over all of master's data?

Has anyone else run into this scenario? I'm surprised by the lack of information when trying to search for this.

Answer 1

Take secondary node off of replica-set and run mongod --repair on it and restart back on replica-set.

This is a common practice which is usually referred to as a "rolling repair". You take each secondary out of the replica set and repair it, and eventually step down the primary for repair as a last step. As long as you always have a majority of your replica set nodes available this approach will minimize potential downtime.

If you are frequently deleting data you should consider using the new PowerOf2Sizes collection option in MongoDB 2.2. This changes the allocation method to allocate document space in powers of two (eg. a 500 byte document would be allocated 512 bytes), which allows for more effective reuse of the space from deleted documents (at the slight expense of a few more bytes per document).

I tried this and couldn't get past permission errors on 'local' collection.

Permission errors on the 'local' collection sound like file system permissions (i.e. based on the user you were running your mongod as). You should run the repair process with the same user.

Shut down secondary node and delete all files in the data directory. Restart mongo and let it recover from master. This actually worked for me but my only concern is, what if your journal collection is full and since it's a capped collection, will you only receive the data that is in the journal or will you actually copy over all of master's data?

It sounds like you are conflating the Journal which is used for durability and crash recovery with the Oplog used for replication.

If you resync a node from the primary, all data will be copied over. During this initial period the node will be in RECOVERING state and is not considered a "healthy" node (i.e. available for queries).

Once the node is caught up it will change to a normal SECONDARY state at which point the oplog will be used for ongoing sync.

Some further reading:

• Replication fundamentals
• Replica set status reference