Why are my MongoDB queries slow after adding new items to the collection?

Question

Any help on this would be massively appreciated.

I have a large Mongodb collection which I store user activities in. It has several compound indexes for the queries I want to run and usually the query performance is fantastic and user activity streams load immediately.

However, recently I have added a background task which adds 20k records to the collection every 2 hours (one new record per user). Since this task was added - I've noticed that the first time I visit my activity stream after these records are added there is a huge delay before the page loads. Then when I refresh the page, it loads quickly.

It almost seems like the new items are only being added to the index after I try to access them. But from what I read on the MongoDB FAQ - they are automatically added to the index - http://docs.mongodb.org/manual/faq/indexes/#should-you-run-ensureindex-after-every-insert. Perhaps it's a case of them just not being added yet?

Here's my collection stats if it helps.

Array
(
[ns] => main.activities
[count] => 26280825
[size] => 3234981772
[avgObjSize] => 123.09285465734
[storageSize] => 4211892224
[numExtents] => 30
[nindexes] => 20
[lastExtentSize] => 844685312
[paddingFactor] => 1.001
[systemFlags] => 1
[userFlags] => 0
[totalIndexSize] => 25240448464
[indexSizes] => Array
    (
        [_id_] => 946551872
        [portfolio_id_1_type_1_timestamp_-1] => 1519746704
        [project_id_1_type_1_timestamp_1] => 1839902512
        [project_id_1] => 1148997808
        [piece_id_1] => 792794016
        [user_id_1_type_1_timestamp_-1] => 1903806128
        [type_1_timestamp_-1] => 1475522720
        [user_id_1_type_1] => 1440243280
        [project_id_1_type_1] => 1394008000
        [project_id_1_type_1_timestamp_1_project_page_timestamp_1] => 2114419888
        [project_id_1_type_1_project_page_timestamp_1] => 1564649296
        [conversation_id_1] => 870670416
        [project_comment_id_1] => 814640288
        [project_comment_id_1_type_1] => 1032408048
        [reply_to_comment_id_1] => 512324512
        [collection_id_1] => 822996160
        [user_id_1] => 1233578528
        [portfolio_id_1] => 852691392
        [type_1_user_id_1] => 1477182448
        [type_1_user_id_-1] => 1483314448
    )

[ok] => 1
)

Answer 1

I think the most likely answer is that the background task is causing documents to fall out of memory as it added the new documents. The second request is fast since you just pulled the documents into memory. It is difficult to tell for sure without more details on the specifics of the documents and queries you are performing.

If you run a mongostat while loading the page the first time and see a number of faults then you need more memory.

You can get back a little memory by removing some of the indexes. Specifically the following indexes can be removed since they are a common prefix of another index:

    [project_id_1] => 1148997808
    [user_id_1_type_1] => 1440243280
    [project_id_1_type_1] => 1394008000
    [project_comment_id_1] => 814640288
    [user_id_1] => 1233578528
    [portfolio_id_1] => 852691392

Also one of these indexes can probably be removed as they are only different in the direction of the last field. The only case that is not true is if your application sorts query results with both { type : 1, user_id : 1} and {type : 1, user_id:-1} as the sort document.

    [type_1_user_id_1] => 1477182448
    [type_1_user_id_-1] => 1483314448

There are other indexes that might be removable depending on the actual queries you are performing.

HTH - Rob.