Is it possible to import data from webserver to MYSQL using sqoop or flume?

Question

I need to import data from webserver to MYSQL database. I wish to know is it feasible with sqoop or flume! I read that import and export of data is possible if either the source or sink is in HDFS, but i wish to know is there any possibility to import data into MYSQL for neither the source nor sink being hdfs? Can anyone guide me with this.. Thanks in advance

Answer 1

Sqoop was specifically designed to load data to and from Hadoop, so it will not help in moving data between a web server and MySQL (using HDFS as an intermediary would most likely be a terrible idea). So I'd say it's not a feasible option.

Flume should be able to plug in perfectly into your webserver (for reading logs, I assume), but there is no native MySQL sink at the moment that I know of (apparent confirmation). It should be reasonably easy to write one; there exists an HBase sink which does something quite similar to what a JDBC sink would need to do, perhaps you could adapt it for your purposes.

Here is a guide to Flume that lists which sinks and sources are available: http://flume.apache.org/FlumeUserGuide.html . The most appropriate source for your web server use case might be a spooling directory source. The guide also contains a short and simple tutorial that walks you through all the steps you need to deploy and configure Flume. If you start from that trivial example, you should be able to incrementally build up a working custom configuration without too much hassle.