How to speed up my mongodb test?
https://stackoverflow.com//questions/9620330
-
09-12-2019 - |
italiano
english
français
española
中国
日本の
العربية
Deutsch
한국어
Português
RussianQuestion
From my test, i get around 500 inserts, 200 queries, 400 updates per sec. I wonder on what i can tweak to increase these numbers.
I have read that others can achieve thousands or even tens of thousand of inserts in their tests, which is far better than my test. I wonder if i am missing something basic?
So, here are the facts :
- I am using a win 32-bit mongodb v2.0.3 with default configuration
- Java driver(2.7.3) with spring mongo, (i dont force fsync)
- Doing a combination of insertion and atomic updates like push, pull, inc, dec, set,
- And repeat all these for 500k times.
- The intention is to simulate user actions like insertion and updates
- No specific indexes are defined, but i think by default there'll be always the unique index on the id ?
- The java app which runs in eclipse IDE is running in the same machine as the mongod server
- H/W spec : core i5, mem 4GB, thinkpad edge
- I notice the java process takes around 280MB and is stable in that number in the course of the loop
The start time is : 2012-03-08 21:50:16
And i monitor using mongostat, and after reaching the time of 22:05:10, i terminate my unfinished application .. and here's the last output of the mongostat
insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn time
499 200 400 0 0 100 0 1023m 1.06g 581m 145 8.5 0 0|0 0|0 645k 97k 3 22:05:01
503 201 403 0 0 102 0 1023m 1.06g 582m 154 10.7 0 0|0 0|1 651k 98k 3 22:05:02
520 208 415 0 0 105 0 1023m 1.06g 582m 176 11.1 0 0|0 0|0 671k 101k 3 22:05:03
504 202 403 0 0 102 0 1023m 1.06g 582m 167 7.2 0 0|0 0|0 651k 98k 3 22:05:04
524 209 419 0 0 106 0 1023m 1.06g 582m 147 8.3 0 0|0 0|0 675k 102k 3 22:05:05
534 213 428 0 0 107 0 1023m 1.06g 583m 176 7.4 0 0|0 0|0 690k 103k 3 22:05:06
531 213 424 0 0 108 0 1023m 1.06g 584m 160 4.9 0 0|0 0|0 685k 104k 3 22:05:07
533 213 427 0 0 107 0 1023m 1.06g 584m 164 6.9 0 0|0 0|0 689k 103k 3 22:05:08
518 208 414 0 0 105 0 1023m 1.06g 585m 158 7.3 0 0|0 0|0 669k 101k 3 22:05:09
521 208 417 0 0 105 0 1023m 1.06g 585m 154 4.7 0 0|0 0|0 673k 101k 3 22:05:10
And then i checked my insertion number :
> db.myCollection.find().size();
90575
And this is an example of my document inserted, which is also updated etc in the process
> db.myCollection.findOne().pretty();
{
"_id" : "b146189a-56a4-4035-8245-c4bd6dc2bd22",
"something1" : "my class is cool !",
"something2" : {
"value" : "this is a statement blah blah",
"name" : "myStatement"
},
"something3" : {
"size" : {
"value" : 0,
"name" : "size"
},
"value" : [
"6810cb0c-fa3e-4ca9-8a27-8432f2d1e828",
"a8276d05-a796-4c43-bc74-edc06d074099"
],
"name" : "myids"
},
"something4" : {
"myattr" : {
"value" : "something",
"name" : "name"
},
"attr" : {
"content" : {
"value" : "another another body body content content",
"name" : "content"
},
"contentId" : "b146189a-56a4-4035-8245-c4bd6dc2bd22",
"name" : "something"
},
"subsubchildchild" : {
"size" : {
"value" : 0,
"name" : "size"
},
"value" : [ ],
"name" : "subBodies"
},
"myId" : "b146189a-56a4-4035-8245-c4bd6dc2bd22",
"name" : "hiccups"
},
"something5" : {
"value" : false,
"name" : "hahaha"
},
"something6" : {
"name" : "okay this is just a test"
},
"something7" : {
"value" : false,
"name" : "remove me !"
},
"something8" : {
"size" : {
"value" : 0,
"name" : "size"
},
"value" : [ ],
"name" : "guess what"
},
"something9" : {
"size" : {
"value" : 0,
"name" : "anotherSize"
},
"value" : [ ],
"name" : "tarantula"
},
"something10" : {
"value" : 8,
"name" : "my exam score"
},
"something11" : {
"size" : {
"value" : 0,
"name" : "justAnotherSize"
},
"value" : [ ],
"name" : "myReference"
},
"something12" : {
"size" : {
"value" : 0,
"name" : "size"
},
"value" : [ ],
"name" : "myOtherReference"
},
"something13" : {
"value" : "8b78fff0-50f5-4992-9972-89f9d944fee7",
"name" : "user"
},
"something14" : {
"dateTime" : "2012-03-08 21:50:17.480000000"
},
"something15" : {
"value" : false,
"name" : "lovely"
}
}
And here's my db stat :
> db.stats();
{
"db" : "qa",
"collections" : 7,
"objects" : 815197,
"avgObjSize" : 622.2093211824872,
"dataSize" : 507223172,
"storageSize" : 610770944,
"numExtents" : 57,
"indexes" : 5,
"indexSize" : 64197952,
"fileSize" : 1056702464,
"nsSizeMB" : 16,
"ok" : 1
}
Also another question from curiosity .. judging from my main collection size, which has around 90k records, and other nonsubstantial collections which shouldnt be big in size, is it reasonable to have around 1TB of fileSize in this case ? Is there anything i can do to help reduce my filesize ?
Please share your thoughts.
Solution
You seem to be getting a lot of faults on the mongostat. Any idea why?
Doing a combination of insertion and atomic updates like push, pull, inc, dec, set
How are you issuing these updates? By _id?
I have read that others can achieve thousands or even tens of thousand of inserts in their tests, which is far better than my test. I wonder if i am missing something basic?
According to mongostat you only have 3 connections active and your lock % is only about 10%.
- Are you multi-threading your inputs?
- Are you ruining this all on the same computer?
- How is system IO?
- Are you doing
WriteConcern.Safe?
These are all considerations that could be affecting your throughput.
is it reasonable to have around 1TB of fileSize in this case ?
Base on your db.stats(), you only have about 600 MB of disk in use.
"storageSize" : 610770944 // = 610,770,944
Your average object size is 622 bytes, but you have 815,197 objects, not the 90k documents you claim to have.
Is there anything i can do to help reduce my filesize ?
Yes, reduce the size of the Keys in your JSON documents. For example:
"something1" : "my class is cool !" => ~28 bytes
"s1": "my class is cool !" => ~20 bytes
Ensure that you are correctly storing the shortened names and let you Data access framework do the lifting of mapping these to more reasonable names.
