Mongodb replica set: Database Size Difference

Question

What are the possible reasons for a difference in database size in primary and secondary nodes of a MongoDB replica set. In my setup, the Secondary node database is of higher size than the primary one. Both nodes have the same number of objects, but the values of "avgObjSize", "dataSize", "storageSize" are higher for secondary node. There is no replication lag as well, as checked from rs.stats()

What can I check?

Answer 1

Brief: Because of different amount of not reclaimed memory space on secondary and different padding factor on secondary and primary.

Long: It could be the case if you have long running primary node where some documents were deleted and inserted, and no compact operation was run. This space would no be reclaimed, and would be counted in dataSize, avgObjSize and storageSize. Secondary could be fully resynced from primary, but only operations from current oplog would be replayed. In this case secondary could have lower values for dataSize, avgObjSize and storageSize. If after that secondary is elected as primary, you could see described difference in sizes. In addition each server has it's own padding factor, that is why you see difference in dataSize.

Concrete scenario could be different, but there are two main causes: amount of not reclaimed memory space and different padding factor.

Answer 2

There's a concept padding factor may be the cause. MongoDB leaves some space for the future updating, so that when the size of object grows, you don't always have to move the object to another storage space.
padding factor can be found in your collections stats:

db.colname.stats()

a sample result:

{
"ns" : "merchant.product",
"count" : 24,
"size" : 23168,
"avgObjSize" : 965.3333333333334,
"storageSize" : 204800,
"numExtents" : 2,
"nindexes" : 1,
"lastExtentSize" : 163840,
"paddingFactor" : 1.0000000000000053,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 8176,
"indexSizes" : {
    "_id_" : 8176
},
"ok" : 1
}

When you update your collection, mongodb changes the value paddingFactor. So there might be slight difference between your 2 nodes because they may not be created at the same time.

When your "padding" doesn't satisfy the new size of your object, mongodb moves it to another storage space. Then the original space is spared for future usage and the object occupies a new block of space. However, this behavior maybe different in your 2 nodes, too because of the different padding factor.

So the size is generally OK.