Changing elasticsearch index's shard-count on the next index-rotation

Question

I have an ELK (Elasticsearch-Kibana) stack wherein the elasticsearch node has the default shard value of 5. Logs are pushed to it in logstash format (logstash-YYYY.MM.DD), which - correct me if I am wrong - are indexed date-wise.

Since I cannot change the shard count of an existing index without reindexing, I want to increase the number of shards to 8 when the next index is created. I figured that the ES-API allows on-the-fly persistent changes.

How do I go about doing this?

Answer 1

You can use the "Template Management" features in Elasticsearch: http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/indices-templates.html

Create a new logstash template by using:

curl -XPUT localhost:9200/_template/logstash -d '
{
  "template": "logstash-*",
  "settings": {
    "number_of_replicas": 1,
    "number_of_shards": 8,
    "index.refresh_interval": "5s"
  },
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true
      },
      "dynamic_templates": [
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "type": "string",
              "index": "analyzed",
              "omit_norms": true,
              "fields": {
                "raw": {
                  "type": "string",
                  "index": "not_analyzed",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      ],
      "properties": {
        "@version": {
          "type": "string",
          "index": "not_analyzed"
        },
        "geoip": {
          "type": "object",
          "dynamic": true,
          "path": "full",
          "properties": {
            "location": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  }
}'

The next time the index that matches your pattern is created, it will be created with your new settings.

Answer 2

The setting is on your elasticsearch. You need to change to config file config/elasticsearch.yml

Change the index.number_of_shards: 8. and restart elasticsearch. The new configuration will set and the new index will use the new configuration, which create 8 shard as you want.

Answer 3

Best would be to use templates and to add one I would recommend Kopf pluin found here: https://github.com/lmenezes/elasticsearch-kopf

You can ofcourse use the API:

curl -XPUT $ELASTICSEARCH-MASTER$:9200/_template/$TEMPLATE-NAME$ -d '$TEMPLATE-CONTENT$'

In the plugin: on the top left corner click on more -> Index templates and then create a new template and make sure you have the following settings as part of your template:

{
  "order": 0,
  "template": "logstash*",
  "settings": {
    "index": {
      "number_of_shards": "5",
      "number_of_replicas": "1"
    }
  },
  "mappings": {### your mapping ####},
  "aliases": {}
}

The above setting will make sure that if a new new index with name logstash* is created it would have 5 number of shards and 1 replica.