MongoDB Index in Memory with Sharding

Question

The word on the street is that MongoDB gets slow if you can't keep the indexes you're using in memory. How does this work with sharding? Does a sharded only keep its own BTree in memory, or does every shard need to keep the index for the entire collection in memory?

Answer 1

Does a sharded only keep its own BTree in memory...?

Yes, each shard manages its own indexes.

The word on the street is that MongoDB gets slow if you can't keep the indexes you're using in memory.

You can actually expect worse when using sharding and secondary indexes. The key problem is that the router process (mongos) knows nothing about data in secondary indexes.

If you do a query using the shard key, it will be routed directly to the correct server(s). In most cases, this levels out the workload. So 100 queries can be spread across 100 servers and each server only answers 1 query.

However, if you do a query using the secondary key, that query has to go to every server. So 100 queries to the router will result 10,000 queries across 100 servers or 100 queries per server. As you add more servers, these "non-shardkey" queries become less and less efficient. The workload does not become more balanced.

Some details are available in the MongoDB docs here.

Answer 2

Just its own portion of the index (it doesn't know about the other shards' data). Scaling wouldn't work very well, otherwise. See this documentation for some more information about sharding: http://www.mongodb.org/display/DOCS/Sharding+Introduction

http://www.mongodb.org/display/DOCS/Choosing+a+Shard+Key