Flume cannot put files to S3 bucket

Question

I'm using flume in its node-only (testing) mode; flume pulls messages from RabbitMQ and puts it to Amazon S3 bucket.

The problem:

Flume actually pulls from RabbitMQ, but files do not appear at S3 bucket.

Technical details:

I start flume in following manner:

flume node -1 -c $FQDN':amqp("exchangeName=[exchange name]", "bindings=[binding name]", "host=127.0.0.1", "port=5672", "userName=[user]", "password=[pass]", "exchangeType=direct", "durableExchange=false", "queueName=[queue name]", "durableQueue=true", "exclusiveQueue=false", "autoDeleteQueue=false", "useMessageTimestamp=true")|collectorSink("s3n://[Amazon key]:[Amazon secret]@[path at S3]","server");' -s "$@" > "$log" 2>&1

Flume log:

After flume restart, when sending something through the relevant exchange and queue, the following lines appear in flume log:

INFO com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink: Opening s3n://[key]:[secret]@[path at S3]

WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded

Answer 1

You will not believe it...

Flume just cannot manage with Amazon key containing '/' The key I used contained one. Then Flume recognized it and just did not called for Amazon.

Solution: Change Amazon key till you get one without '/'