How to MapReduce on Mongo replica using the Perl MongoDB package

Question

I am using Perl v5.12 with the MongoDB package v0.45.

I would like to run a MapReduce job to create a new collection that I will then create a cursor for later. My other desire is that this job runs on a replica and not on the master.

As defined in the perl doc, MapReduce jobs are to be executed using the run_command method. When I execute the perl script, I get:

Mongo error: not master at perlib/Connections.pm line 63.

After reading the MongoDB documentation on CPAN, there seems only to be a method for enabling cursors to read from replicas. So that method doesn't apply to calls to run_command().

Here's my code:

sub get_data {
    my $self = shift;
    my $dbh = shift;
    my $collection_h = shift;   
    my $since_time = $self->get_date_time(shift);
    my $loop_limit = $self->get_data_limit(shift);

    my %data = ();
    my $ctr = 0;

    my $temp_collection='temp_collection';

    my $ids = $dbh->run_command([
         "mapreduce" => $collection_h->{'name'}
        ,"map" => _get_map()
        ,"reduce" => _get_reduce()
        ,"query"    =>  {'to.id' => {'$exists'=>'true'}, 'updatedDate' => { '$gte' => $since_time }}
        ,"out" => $temp_collection      
    ]);

    die ("Mongo error: $ids") unless ref($ids) eq 'HASH';

        # next we create a cursor to this new collection
    my $cfs_h = $dbh->$temp_collection;
    my $id_cursor = $cfs_h->find()->limit($loop_limit);

    $id_cursor->slave_okay(1);
    $id_cursor->immortal(1);
    ...
}

sub _get_map {
    return "function() {emit(this.to.id, 1);}"; 
}

sub _get_reduce {

    return "function(k,vals) {return 1;}"
}

Has anyone encountered this problem of trying to use MapReduce on a replica? Have you done it successfully? If so, could you please share how you did it.

Answer 1

When performing a Map/Reduce operation on a secondary, the results must be returned inline. The output must be returned with the "results" object and cannot be written to disk, because secondaries cannot be written to. When the results are output in this fashion, there is a limit of 16MB for the results set. There is a note on this under the "{ inline : 1}" bullet in the "Output options" section of the Map Reduce documentation. http://www.mongodb.org/display/DOCS/MapReduce#MapReduce-Outputoptions