Implementing Elasticsearch on existing MySQL database on Rails

Question

I am developing a web application using Ruby on Rails with MySQL database. I would like to include elasticsearch functionality within the application. Do I have to migrate the MySQL database to a NoSQL database and configure the database connectivity (start from scratch) or is there a way that I can achieve it in a better way?

Answer 1

You do need to get your data into Elasticsearch ("index" the data) for it to be usable from Elasticsearch searches.

The new way to do this is to use the Logstash JDBC Input to effectively do the same thing as the JDBC River below.

---

The old way to do this was to use the JDBC River architecture. This functionality was removed from Elasticsearch 2.x because Rivers in general introduce a lot of unpredictable variability to Elasticsearch (basically a small application is running inside of it that can balloon in memory). The JDBC River was particularly well done, but no new development should begin using it due to the removal. Doing so can be a matter of you doing it manually, or you can do it automatically with the JDBC River plugin, which is widely used for exactly this purpose.

Ideally, you can get away with the simple example that maps the column names as keys to their values (adding a period inside the name, like name.first would create object nesting).

curl -XPUT 'localhost:9200/_river/my_jdbc_river/_meta' -d '{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://localhost:3306/test",
        "user" : "",
        "password" : "",
        "sql" : "select products.name as \"product.name\", orders.customer as \"product.customer.name\", orders.quantity * products.price as \"product.customer.bill\" from products, orders where products.name = orders.product"
    }
}'

That is directly from the documentation, which results in results like the following:

mysql> select products.name as "product.name", orders.customer as "product.customer", orders.quantity * products.price as "product.customer.bill" from products, orders where products.name = orders.product ;
+--------------+------------------+-----------------------+
| product.name | product.customer | product.customer.bill |
+--------------+------------------+-----------------------+
| Apples       | Big              |                     1 |
| Bananas      | Large            |                     2 |
| Oranges      | Huge             |                     6 |
| Apples       | Good             |                     2 |
| Oranges      | Bad              |                     9 |
+--------------+------------------+-----------------------+
5 rows in set, 5 warnings (0.00 sec)

Finally, the JDBC River will translate those results into:

id=0 {"product":{"name":"Apples","customer":{"bill":1.0,"name":"Big"}}}
id=1 {"product":{"name":"Bananas","customer":{"bill":2.0,"name":"Large"}}}
id=2 {"product":{"name":"Oranges","customer":{"bill":6.0,"name":"Huge"}}}
id=3 {"product":{"name":"Apples","customer":{"bill":2.0,"name":"Good"}}}
id=4 {"product":{"name":"Oranges","customer":{"bill":9.0,"name":"Bad"}}}

All of the example information was taken directly from the documentation.