How to make MongoDB return actual data to other readers after acknowledged write?

Question

I am trying to emulate some kind of "transaction" in MongoDB. I'm locking critical code section (using memcache), reading document from MongoDB, changing it a bit, writing it back and unlocking section. However rarely such operations if run concurrent can work not as expected. After acknowledged (w = 1) write another thread can still read previous version of the document. This happens roughly 1/1000 times under heavy load (10+ concurrent requests) but ruins whole idea.

I tried to use fsync option, but it makes write too slow to be usable (these operations should be run oftenly). journaled option (j = 1) does not solve the issue. Is there any way to make other threads reading consistent version of document after acknowledged write from some thread?

Here is example code (I changed it to simple counter, the problem can be reproduced anyway):

$mchost = "127.0.0.1";
$mcport = 11211;
$lockkey = "testlockkey";

$mongoconnstr = "mongodb://localhost";
$mongodb = "testdb";
$mongotbl = "test";
$mongoopts = array("w" => 1);
$varname = "counter";

function LIBLockSection($id, $timeout)
{
    for ($i = 0; $i < $timeout; $i++) {
        if (memcache_add($GLOBALS["mc"], $id, 1, FALSE, 60)) return 1;
        usleep(50000);
    }
    return 0;
}

function LIBUnlockSection($id)
{
    return memcache_delete($GLOBALS["mc"], $id);
}

function MNGGetTable($connstr, $dbname, $tname)
{
    $m = new Mongo($connstr);
    $db = $m->selectDB($dbname);
    return $db->selectCollection($tname);
}

$GLOBALS["mc"] = memcache_connect($mchost, $mcport) or die('memcached is down!');
$tbl = MNGGetTable($mongoconnstr, $mongodb, $mongotbl);

$lockres = LIBLockSection($lockkey, 50);
if (!$lockres) {
    exit;
}

$x = $tbl->findOne();
if (!isset($x[$varname])) $x[$varname] = 0;
$x[$varname]++;
$opts = $mongoopts;
$opts["upsert"] = TRUE;
$tbl->update(array(), array($varname => $x[$varname]), $opts);

LIBUnlockSection($lockkey);
trigger_error($x[$varname]);

This script is writting its output to standard PHP error log. To run it I am using ab (for test.php in the root of the server):

ab -c 10 -n 10000 http://127.0.0.1/test.php

Expected output is sequential numbers from 1 to 10000. But without fsync option sometimes we get doubles:

[Tue Oct 01 00:53:53 2013] [error] [client 127.0.0.1] PHP Notice:  344 in /var/www/test.php on line 49
[Tue Oct 01 00:53:53 2013] [error] [client 127.0.0.1] PHP Notice:  344 in /var/www/test.php on line 49

Right now I see only one possible solution: using "version" variable in document and findAndModify operation on DB to ensure the document is updated only if the version was not changed between operations. But the code will look ugly so I am wondering if there is any other way to solve this.

Answer 1

I see a few problems with your code that explains:

But without fsync option sometimes we get doubles:

You are for one, using the deprecated Mongo class:

$m = new Mongo($connstr);

The default write concern of this is socket acknowledge.

Normally this would be countered by:

$mongoopts = array("w" => 1);

But w is not compatible with Mongo, it is instead compatible with MongoClient. The option for Mongo is safe.

That means you are never actually getting an acknowledgement of a successful OP from MongoDB, instead just getting an acknowledgement from the socket.

Try fixing your code and seeing if the problem comes back.

However this link might help: http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/ to get atomic, race condition free incrementing keys from MongoDB.