latency with group in pymongo in tests

Question

Good Day.

I have faced following issue using pymongo==2.1.1 in python2.7 with mongo 2.4.8

I have tried to find solution using google and stack overflow but failed.

What's the issue?

I have following function

from bson.code import Code
def read(groupped_by=None):
    reducer = Code("""
           function(obj, prev){
             prev.count++;
           }
           """)

    client = Connection('localhost', 27017)
    db = client.urlstats_database

    results = db.http_requests.group(key={k:1 for k in groupped_by},
                                     condition={},
                                     initial={"count": 0},
                                     reduce=reducer)
    groupped_by = list(groupped_by) + ['count']
    result = [tuple(res[col] for col in groupped_by) for res in results]
    return sorted(result)

Then I am trying to write test for this function

class UrlstatsViewsTestCase(TestCase):
    test_data = {'data%s' % i : 'data%s' % i for i in range(6)}

    def test_one_criterium(self):
        client = Connection('localhost', 27017)
        db = client.urlstats_database
        for column in self.test_data:
            db.http_requests.remove()
            db.http_requests.insert(self.test_data)

            response = read([column])
            self.assertEqual(response, [(self.test_data[column], 1)])

this test sometimes fails as I understand because of latency. As I can see response has not cleaned data in it

If I add delay after remove test pass all the time.

Is there any proper way to test such functionality?

Thanks in Advance.

Answer 1

Thanks All.

There is no MongoClient class in version of pymongo I use. So I was forced to find out what exactly differs.

As soon as I upgrade to 2.2+ I will test whether everything is ok with MongoClient. But as for connection class one can use write concern to control this latency.

I older version One should create connection with corresponding arguments.

I have tried these twojournal=True, safe=True (journal write concern can't be used in non-safe mode)

j or journal: Block until write operations have been commited to the journal. Ignored if the server is running without journaling. Implies safe=True.

I think this make performance worse but for automatic tests this should be ok.

Answer 2

A few questions regarding your environment / code:

• What version of pymongo are you using?
• If you are using any of the newer versions that have MongoClient, is there any specific reason you are using Connection instead of MongoClient?

The reason I ask second question is because Connection provides fire-and-forget kind of functionality for the operations that you are doing while MongoClient works by default in safe mode and is also preferred approach of use since mongodb 2.2+.

The behviour that you see is very conclusive for Connection usage instead of MongoClient. While using Connection your remove is sent to server, and the moment it is sent from client side, your program execution moves to next step which is to add new entries. Based on latency / remove operation completion time, these are going to be conflicting as you have already noticed in your test case.

Can you change to use MongoClient and see if that helps you with your test code?

Additional Ref: pymongo: MongoClient or Connection