Routing elasticsearch document indexing issue

Question

While creating an index of books with several authors but belonging only to a subset of 11 different publications, I intended to have 11 shards of the index on a single node.

Obviously the routing part of my mapping looked like,

"book":{
      "_routing":{
         "required":true,
         "path":"publication"
      },....

publication itself is a string which could be of 11 different types.

The settings part of my mapping looks like

body: {
    settings: {
       index: {
           number_of_shards: 11,
           number_of_replicas: 1
        }
     },....  

The problem I'm facing is that docs of different publication have been indexed in the same shard while some shards are lying empty after I looked at the results of books/_status andbooks/_segments queries.

I looked into the reroute API but that obviously should be looked for other issues.

This unequal distribution of docs after specifying routing is tempting me to ditch routing altogether.

The version I've been using is 0.90.5.

Answer 1

Routing doesnt gaurantee that all the items on a shard will have the same routing key, it will make sure that all the docs that share a routing key are all on the same shard.

Here is a good ES post about it

And a quote from the bottom:

This shard almost certainly has documents other than those belonging to user123. Routing ensures that documents of a particular routing value all go to the same shard…but that doesn’t mean that other documents aren’t routed to the shard too.