I have a Flume agent writing tweets to a HBase sink.
After a few seconds, transactions to the sink are failing and every 8-10 seconds I get these error messages in the Flume agent log telling me the transaction to HBase is failing.
The strange thing is that some tweets still get through and go into the HBase table. What could be causing this?
This is running on a single node Cloudera Quickstart VM, could it be a problem with resources?
This is the agent log
9:20:44.618 PM ERROR org.apache.flume.SinkRunner
Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Could not write events to Hbase. Transaction failed, and rolled back.
at org.apache.flume.sink.hbase.AsyncHBaseSink.process(AsyncHBaseSink.java:245)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
9:20:53.883 PM ERROR org.apache.flume.SinkRunner
Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Could not write events to Hbase. Transaction failed, and rolled back.
at org.apache.flume.sink.hbase.AsyncHBaseSink.process(AsyncHBaseSink.java:245)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
These are some strange things in the debug log, maybe related?
2014-03-06 09:39:12,069 DEBUG org.apache.zookeeper.client.ZooKeeperSaslClient: Could not retrieve login configuration: java.lang.SecurityException: Unable to locate a login configuration
2014-03-06 09:39:12,298 DEBUG org.apache.zookeeper.ClientCnxn: An exception was thrown while closing send thread for session 0x144965080900029 : Unable to read additional data from server sessionid 0x144965080900029, likely server has closed socket
This is my agent configuration
TwitterAgent.sinks.HBaseTweet.channel = MemChannel
TwitterAgent.sinks.HBaseTweet.type = org.apache.flume.sink.hbase.AsyncHBaseSink
TwitterAgent.sinks.HBaseTweet.table = tweets
TwitterAgent.sinks.HBaseTweet.columnFamily = tweet
TwitterAgent.sinks.HBaseTweet.batchSize = 100
TwitterAgent.sinks.HBaseTweet.serializer = flume_hdfs.hbase.util.AsyncHbaseTwitterEventSerializer
TwitterAgent.sinks.HBaseTweet.serializer.columns = tweet:id,tweet:created_at,tweet:source,tweet:favourited,tweet:text
TwitterAgent.sinks.HBaseTweet.serializer.delimiter = \\t
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 200
TwitterAgent.channels.MemChannel.transactionCapacity = 100
Some metrics from the log when stopping the agent, might be interesting
Component type: CHANNEL, name: MemChannel stopped
Shutdown Metric for type: CHANNEL, name: MemChannel. channel.start.time == 1394093630078
Shutdown Metric for type: CHANNEL, name: MemChannel. channel.stop.time == 1394093894804
Shutdown Metric for type: CHANNEL, name: MemChannel. channel.capacity == 200
Shutdown Metric for type: CHANNEL, name: MemChannel. channel.current.size == 125
Shutdown Metric for type: CHANNEL, name: MemChannel. channel.event.put.attempt == 220
Shutdown Metric for type: CHANNEL, name: MemChannel. channel.event.put.success == 209
Shutdown Metric for type: CHANNEL, name: MemChannel. channel.event.take.attempt == 3059
Shutdown Metric for type: CHANNEL, name: MemChannel. channel.event.take.success == 9
Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Could not write events to Hbase. Transaction failed, and rolled back.
at org.apache.flume.sink.hbase.AsyncHBaseSink.process(AsyncHBaseSink.java:245)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Component type: SINK, name: HBaseTweet stopped
Shutdown Metric for type: SINK, name: HBaseTweet. sink.start.time == 1394093630407
Shutdown Metric for type: SINK, name: HBaseTweet. sink.stop.time == 1394093894833
Shutdown Metric for type: SINK, name: HBaseTweet. sink.batch.complete == 27
Shutdown Metric for type: SINK, name: HBaseTweet. sink.batch.empty == 0
Shutdown Metric for type: SINK, name: HBaseTweet. sink.batch.underflow == 7
Shutdown Metric for type: SINK, name: HBaseTweet. sink.connection.closed.count == 1
Shutdown Metric for type: SINK, name: HBaseTweet. sink.connection.creation.count == 1
Shutdown Metric for type: SINK, name: HBaseTweet. sink.connection.failed.count == 0
Shutdown Metric for type: SINK, name: HBaseTweet. sink.event.drain.attempt == 3053
Shutdown Metric for type: SINK, name: HBaseTweet. sink.event.drain.sucess == 9
HBase Regionserver error
2014-03-08 09:37:44,371 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family retweet does not exist in region tweets,,1394029330397.953f602dd0790637df8106720396f219. in table 'tweets', {NAME => 'entities', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', ENCODE_ON_DISK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'retweeted_status', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', ENCODE_ON_DISK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'tweet', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', ENCODE_ON_DISK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'user', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', ENCODE_ON_DISK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
at org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:5475)
at org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:3022)
at org.apache.hadoop.hbase.regionserver.HRegion.internalPut(HRegion.java:2900)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:2083)
at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:2239)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1428)