How to upgrade a running Elasticsearch older instance to a newer version?

Question

Essentially I cannot find documents or resources that explains the procedure of upgrading a running Elasticsearch instance into the current version.

Please help me out in a few scenarios:

1. If I am running an Elasticsearch instance in a single server, how do I upgrade the instance and not lose data?

2. If I am running multiple Elasticsearch instances in a number of servers, how do I keep my operations running, while I upgrade my Elasticsearch instances without losing data?

If there are proper procedures or explanations on this it will greatly help my understanding and work. Thanks!

Answer 1

1. All node data is stored in elasticsearch data directory. It's data/cluster_name/nodes by default in elasticsearch home. So, in general, as long as data directory is preserved and config files in the new version are compatible with the old version, the new instance should have the same data as the old one. Please note that some releases have special additional requirements outlined in release notes. For example, upgrade to 0.19 from 0.18 requires issuing a full flush of all the indices in the cluster.

2. There is really no good way to accomplish this. Nodes are communicating using binary protocol that is not backward compatible. So, if protocol in the new version changes, old nodes and new nodes are unable to understand each other. Sometimes it's possible to mix nodes with different minor versions within the same cluster and do rolling upgrade. However, as far as I understand, there is no explicit guarantee on compatibility between nodes even within minor releases and major releases always require require full cluster restart. If downtime during full cluster restart is not an option, a nice technique by DrTech might be a solution.

Answer 2

There is a lot more information about ElasticSearch upgrade these days than it used to be.

Here are my usual steps when upgrading ElasticSearch:

1. Backup the data : Snapshot and Restore

2. Upgrade Guide : Upgrading ElasticSearch

The main idea is that you shut down one instance of the ES cluster at a time, upgrade the ES version on that instance node, and bring it up again so it can join back the cluster.

In brief, here are the important steps:

1. Disable Shard reallocation

   curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "cluster.routing.allocation.enable" : "none" } }'

2. Shutdown the instance:

   curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'

3. Install the new ElasticSearch version on the host and Start it.

4. Enable shard re-allocation:

   curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "cluster.routing.allocation.enable" : "all" } }'

5. Watch cluster go from yellow state to green with:

curl -X GET http://localhost:9200/_cat/health?v // monitors the overal cluster state

curl -X GET http://localhost:9200/_cat/nodes?v // verify that the new node joined the cluster

curl -X GET http://localhost:9200/_cat/shards?v // see shards being started, initialized and relocated

6. Repeat for the next node.

In the terms of ordering, update first the master nodes, then data nodes, then load-balancing/client nodes.

Answer 3

It should be worth mentioning that there is now documentation on doing this upgrade but it doesn't rank very highly on search results:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html

as well as a breaking changes document:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/breaking-changes.html

Answer 4

The other answers are a little dated based off what exists now, so this new information will supplement upgrading newer versions of Elasticsearch, like upgrading 6.x to 7.x, or 5.x to 6.x. There are two main options for an Elasticsearch upgrade: rolling upgrade or full cluster restart.

The rolling upgrade allows nodes to be upgraded one at a time so service is not interrupted. On the other hand, the full cluster restart requires every node to be shut down, upgraded, and then brought back up. This means there will be downtime between upgrades that must be accounted for.

This is much easier to do compared to a few years ago when the only viable option was a snapshot & restore.

Answer 5

If you are running Ubuntu or Debian based Linux, here is an Ansible script to do a rolling upgrade, as long as you are not upgrading between major versions.

Minor versions such as 1.3 -> 1.4.3 are ok

Major versions such as 0.8 -> 1.4.3 will not work.

https://github.com/ekhoinc/ansible-examples/blob/master/elasticsearch-rolling-upgrade.yml

It could easily be modified to work with RHEL based linux (just 2 lines to change)