Hadoop Flume java client to transfer log to hdfs

Question

Is it possible to transfer web server logs to HDFS using flume java client?? If so, how?? please do help me.....

I've seen several people have problems writing to HDFS

Answer 1

Use the log4j Flume appender to send the application events to Flume. And then Flume can write the events to HDFS, which can be later analyzed using Hive or some other framework. Check this article for detailed explanation with the code and configuration files. The appender mentioned in the article is part of Flume and has been moved to Log4J 2.

Also, take a look at Scribe from Facebook. But, it has not been updated for more than 2 years. Looks like it is not gathering much attention.