MongoDB Maintenance

Question

I am writing a Maintenance Script for MongoDB that will compact collections from a replica set on a scheduled basis. The script I have so far looks good, doing the job it is expected to do against Primary and Secondary nodes. There is a problem, however:

The system that is using the MongoDB Replica Set is a high-availability web-queue, which is constantly being written to and read from. Thus even the several seconds downtime when calling rs.StepDown() are absolutely unacceptable. Is there a way to safely demote a Primary node without MongoExceptions from the hundreds of clients?

Thanks!

p.s. here is the actual version of the script that should be run via a cron-job at low-load times once a month

// assuming we are a replica set ;-)
if(rs.isMaster().setName){
    try {
        //check if the script is run against a master
        if(rs.isMaster().ismaster){ //if so, step down as master        
            print("Connected to a PRIMARY node")
            print("Will try to step down as primary");
            rs.stepDown();

            // after stepdown connections are dropped. do an operation to cause reconnect:
            rs.isMaster();            

            // now ready to go.      
            // wait for another node to become primary -- it may need data from us for the last 
            // small sliver of time, and if we are already compacting it cannot get it while the 
            // compaction is running.            
            var counter = 1;
            while( 1 ) { 
                var m = rs.isMaster();
                if( m.ismaster ) {
                    print("ERROR: no one took over during our stepDown duration. we are primary again!");
                    assert(false);
                }

                if( m.primary ){ // someone else is, great
                    print("new master host is: "+m.primary);
                    break; 
                }
                print("waiting "+counter+" seconds");
                sleep(1000);
                counter++;
            }
        }else{
            print("Connected to a SECONDARY node");
            print("Going into Recovery Mode and Compacting");            
        }

        // someone else is primary, so we are ready to proceed with a compaction
        print(" ");
        print("Compacting...");
        print("- queue");
        printjson( db.runCommand({compact:"queue"}) );
        print(" ");

    } catch(e) { 
        print(" ");
        print("ACHTUNG! Exception:" + e);
        print(" ");
    }
}else{
    print('This script works with replica sets only');
}

Answer 1

Is there a way to safely demote a Primary node without MongoExceptions from the hundreds of clients?

No. MongoDB does not have any form of "anticipatory change-over" or "maintenance change-over".

Doing this cycle of compactions is the correct thing to do. But you will have to wait for a maintenance window to compact the primary.

... is a high-availability web-queue.

Are you using Capped Collections for this? Capped collections don't need compacting. The objects in a capped collection cannot be resized, but that is typically not a problem for Queue objects.

Though not an ideal solution it may solve your problem.