Cassandra crashes, no idea whats going wrong

Question

The system crashes after sometime with following exceptions:

Python Pycassa Exception reads:

Error: An attempt was made to connect to each of the servers twice, but none of the attempts succeeded. The last failure was TTransportException: Could not connect to 10.242.253.141:9160

Cassandra system.log

 WARN [MemoryMeter:1] 2012-07-31 12:44:17,555 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.16881217597040166
 INFO [MemoryMeter:1] 2012-07-31 12:44:17,555 Memtable.java (line 213) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is 1.0 (just-counted was 1.0).  calculation took 28ms for 122 columns
 WARN [MemoryMeter:1] 2012-07-31 12:44:22,583 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.11368422493368459
 INFO [MemoryMeter:1] 2012-07-31 12:44:22,586 Memtable.java (line 213) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is 1.0 (just-counted was 1.0).  calculation took 39ms for 358 columns
 INFO [GossipTasks:1] 2012-07-31 12:44:22,609 Gossiper.java (line 811) InetAddress /10.204.81.77 is now dead.
 WARN [MemoryMeter:1] 2012-07-31 12:44:25,244 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.13325571841646536
 INFO [MemoryMeter:1] 2012-07-31 12:44:25,244 Memtable.java (line 213) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is 1.0 (just-counted was 1.0).  calculation took 46ms for 484 columns
 INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,679 CassandraDaemon.java (line 218) Stop listening to thrift clients
 INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,762 MessagingService.java (line 539) Waiting for messaging service to quiesce
 INFO [ACCEPT-/10.242.253.141] 2012-07-31 12:44:26,780 MessagingService.java (line 695) MessagingService shutting down server thread.
ERROR [Thrift:40765] 2012-07-31 12:44:26,858 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
    at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
    at org.apache.cassandra.service.StorageProxy.scheduleLocalHint(StorageProxy.java:386)
    at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:339)
    at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:119)
    at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:260)
    at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:193)
    at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:637)
    at org.apache.cassandra.thrift.CassandraServer.internal_insert(CassandraServer.java:511)
    at org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:519)
    at org.apache.cassandra.thrift.Cassandra$Processor$insert.getResult(Cassandra.java:3016)
    at org.apache.cassandra.thrift.Cassandra$Processor$insert.getResult(Cassandra.java:3004)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
ERROR [Thrift:42739] 2012-07-31 12:44:26,935 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
    at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
    at org.apache.cassandra.service.StorageProxy.scheduleLocalHint(StorageProxy.java:386)
    at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:339)
    at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:119)
    at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:260)
    at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:193)
    at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:637)
    at org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:587)
    at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:595)
    at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3112)
    at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3100)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

cassanda.yaml

cluster_name: "Test Cluster"

initial_token:

hinted_handoff_enabled: true
max_hint_window_in_ms: 3600000 # one hour
hinted_handoff_throttle_delay_in_ms: 1

authenticator: org.apache.cassandra.auth.AllowAllAuthenticator

authority: org.apache.cassandra.auth.AllowAllAuthority

partitioner: org.apache.cassandra.dht.RandomPartitioner

data_file_directories:
    - /mnt
    - /ebs

commitlog_directory: /mnt/cassandra_log/

key_cache_size_in_mb:

key_cache_save_period: 14400


row_cache_size_in_mb: 0

row_cache_save_period: 0


row_cache_provider: SerializingCacheProvider

saved_caches_directory: /var/lib/cassandra/saved_caches

commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000

seed_provider:
    # Addresses of hosts that are deemed contact points. 
    # Cassandra nodes use this list of hosts to find each other and learn
    # the topology of the ring.  You must change this if you are running
    # multiple nodes!
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          # seeds is actually a comma-delimited list of addresses.
          # Ex: "<ip1>,<ip2>,<ip3>"
          - seeds: "127.0.0.1"

flush_largest_memtables_at: 0.45

reduce_cache_sizes_at: 0.45
reduce_cache_capacity_to: 0.45

concurrent_reads: 8
concurrent_writes: 8

memtable_total_space_in_mb: 64

commitlog_total_space_in_mb: 64

memtable_flush_writers: 1

memtable_flush_queue_size: 1

trickle_fsync: false
trickle_fsync_interval_in_kb: 10240

storage_port: 7776

ssl_storage_port: 7001

listen_address: 10.242.253.141


rpc_address: 0.0.0.0
rpc_port: 9160

rpc_keepalive: true

rpc_server_type: sync



thrift_framed_transport_size_in_mb: 15

thrift_max_message_length_in_mb: 16

incremental_backups: false

snapshot_before_compaction: false

auto_snapshot: true

column_index_size_in_kb: 64

in_memory_compaction_limit_in_mb: 64

multithreaded_compaction: false

compaction_throughput_mb_per_sec: 16

compaction_preheat_key_cache: true

rpc_timeout_in_ms: 10000

endpoint_snitch: SimpleSnitch

dynamic_snitch_update_interval_in_ms: 100 
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1

request_scheduler: org.apache.cassandra.scheduler.NoScheduler

index_interval: 128

encryption_options:
    internode_encryption: none
    keystore: conf/.keystore
    keystore_password: cassandra
    truststore: conf/.truststore
    truststore_password: cassandra
    # More advanced defaults below:
    # protocol: TLS
    # algorithm: SunX509
    # store_type: JKS
    # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA]

So whats going wrong? Please guide?

Answer 1

I had the same problem. I figured out two things first go to

sudo vim /etc/cassandra/cassandra.yaml

and change the listen ip to the private ip is you are on AWS and change the rpc_ip to 0.0.0.0

also make sure your server has at least 2GB RAM or cassandra will keep crashing!!

to start cassandra on ubuntu try this command:

sudo service cassandra restart

and to check the status do

sudo service cassandra status

Hope this helps

Answer 2

It looks like the Cassandra process is being shut down, presumably from being sent a signal. These lines are what indicate that:

INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,679 CassandraDaemon.java (line 218) Stop listening to thrift clients
INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,762 MessagingService.java (line 539) Waiting for messaging service to quiesce
INFO [ACCEPT-/10.242.253.141] 2012-07-31 12:44:26,780 MessagingService.java (line 695) MessagingService shutting down server thread.
ERROR [Thrift:40765] 2012-07-31 12:44:26,858 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down