ES to Crate data transfer

Question

I want to migrate the data from ES (1.0.0) to Crate (currently available version 0.26.0) to check it out, how could I do it? What should I do with indexes, does ES index fit Crate?

One might think of several flows:

1. Smooth transition, e.g. when I do hybrid cluster with ES + Crate nodes, with replica at least one, and afterwards kill ES nodes one by one.
2. We're doing backups via 'elasticsearch-cloud-aws' plugin is it possible to 'restore' this snapshots, but having Crate nodes instead of native ES ones?

Any other variants to transfer big chunk of data, say 1TB?

Thanks.

Answer 1

Crate is using ES 1.0.1 since 0.24.0 so the lucene indices are compatible.

This means that the "smooth transition" flow you described should work. (But I haven't actually tried it).

But as long as there are non-crate nodes in the cluster the SQL queries on crate won't work because regular ES nodes don't know how to handle the requests.

Some more things you should probably aware of as it could cause troubles:

Crate introduced explicit array types with 0.25.0. This means that If you create a new table with e.g. create table test (id integer ...) the id field will use the new doc_values field format that was added to lucene some time ago. If you want to use arrays you have to create the tables using SQL (with type definitions like array(integer) ).

If you try to query data that was inserted using elasticsearch you might get strange results or even run into errors if the fields contain arrays but are missing the declaration in the schema definition.

Hope this helps a bit. Unfortunately I can't give you an answer regarding the elasticsearch-cloud-aws plugin, but maybe someone else can.