Cassandra crashes, no idea whats going wrong

StackOverflow https://stackoverflow.com/questions/11751993

  •  24-06-2021
  •  | 
  •  

Question

The system crashes after sometime with following exceptions:

Python Pycassa Exception reads:

Error: An attempt was made to connect to each of the servers twice, but none of the attempts succeeded. The last failure was TTransportException: Could not connect to 10.242.253.141:9160

Cassandra system.log

 WARN [MemoryMeter:1] 2012-07-31 12:44:17,555 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.16881217597040166
 INFO [MemoryMeter:1] 2012-07-31 12:44:17,555 Memtable.java (line 213) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is 1.0 (just-counted was 1.0).  calculation took 28ms for 122 columns
 WARN [MemoryMeter:1] 2012-07-31 12:44:22,583 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.11368422493368459
 INFO [MemoryMeter:1] 2012-07-31 12:44:22,586 Memtable.java (line 213) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is 1.0 (just-counted was 1.0).  calculation took 39ms for 358 columns
 INFO [GossipTasks:1] 2012-07-31 12:44:22,609 Gossiper.java (line 811) InetAddress /10.204.81.77 is now dead.
 WARN [MemoryMeter:1] 2012-07-31 12:44:25,244 Memtable.java (line 197) setting live ratio to minimum of 1.0 instead of 0.13325571841646536
 INFO [MemoryMeter:1] 2012-07-31 12:44:25,244 Memtable.java (line 213) CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is 1.0 (just-counted was 1.0).  calculation took 46ms for 484 columns
 INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,679 CassandraDaemon.java (line 218) Stop listening to thrift clients
 INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,762 MessagingService.java (line 539) Waiting for messaging service to quiesce
 INFO [ACCEPT-/10.242.253.141] 2012-07-31 12:44:26,780 MessagingService.java (line 695) MessagingService shutting down server thread.
ERROR [Thrift:40765] 2012-07-31 12:44:26,858 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
    at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
    at org.apache.cassandra.service.StorageProxy.scheduleLocalHint(StorageProxy.java:386)
    at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:339)
    at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:119)
    at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:260)
    at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:193)
    at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:637)
    at org.apache.cassandra.thrift.CassandraServer.internal_insert(CassandraServer.java:511)
    at org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:519)
    at org.apache.cassandra.thrift.Cassandra$Processor$insert.getResult(Cassandra.java:3016)
    at org.apache.cassandra.thrift.Cassandra$Processor$insert.getResult(Cassandra.java:3004)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
ERROR [Thrift:42739] 2012-07-31 12:44:26,935 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
    at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
    at org.apache.cassandra.service.StorageProxy.scheduleLocalHint(StorageProxy.java:386)
    at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:339)
    at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:119)
    at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:260)
    at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:193)
    at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:637)
    at org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:587)
    at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:595)
    at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3112)
    at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3100)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

cassanda.yaml

cluster_name: "Test Cluster"

initial_token:

hinted_handoff_enabled: true
max_hint_window_in_ms: 3600000 # one hour
hinted_handoff_throttle_delay_in_ms: 1

authenticator: org.apache.cassandra.auth.AllowAllAuthenticator

authority: org.apache.cassandra.auth.AllowAllAuthority

partitioner: org.apache.cassandra.dht.RandomPartitioner

data_file_directories:
    - /mnt
    - /ebs

commitlog_directory: /mnt/cassandra_log/

key_cache_size_in_mb:

key_cache_save_period: 14400


row_cache_size_in_mb: 0

row_cache_save_period: 0


row_cache_provider: SerializingCacheProvider

saved_caches_directory: /var/lib/cassandra/saved_caches

commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000

seed_provider:
    # Addresses of hosts that are deemed contact points. 
    # Cassandra nodes use this list of hosts to find each other and learn
    # the topology of the ring.  You must change this if you are running
    # multiple nodes!
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          # seeds is actually a comma-delimited list of addresses.
          # Ex: "<ip1>,<ip2>,<ip3>"
          - seeds: "127.0.0.1"

flush_largest_memtables_at: 0.45

reduce_cache_sizes_at: 0.45
reduce_cache_capacity_to: 0.45

concurrent_reads: 8
concurrent_writes: 8

memtable_total_space_in_mb: 64

commitlog_total_space_in_mb: 64

memtable_flush_writers: 1

memtable_flush_queue_size: 1

trickle_fsync: false
trickle_fsync_interval_in_kb: 10240

storage_port: 7776

ssl_storage_port: 7001

listen_address: 10.242.253.141


rpc_address: 0.0.0.0
rpc_port: 9160

rpc_keepalive: true

rpc_server_type: sync



thrift_framed_transport_size_in_mb: 15

thrift_max_message_length_in_mb: 16

incremental_backups: false

snapshot_before_compaction: false

auto_snapshot: true

column_index_size_in_kb: 64

in_memory_compaction_limit_in_mb: 64

multithreaded_compaction: false

compaction_throughput_mb_per_sec: 16

compaction_preheat_key_cache: true

rpc_timeout_in_ms: 10000

endpoint_snitch: SimpleSnitch

dynamic_snitch_update_interval_in_ms: 100 
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1

request_scheduler: org.apache.cassandra.scheduler.NoScheduler

index_interval: 128

encryption_options:
    internode_encryption: none
    keystore: conf/.keystore
    keystore_password: cassandra
    truststore: conf/.truststore
    truststore_password: cassandra
    # More advanced defaults below:
    # protocol: TLS
    # algorithm: SunX509
    # store_type: JKS
    # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA]

So whats going wrong? Please guide?

Was it helpful?

Solution

I had the same problem. I figured out two things first go to

sudo vim /etc/cassandra/cassandra.yaml

and change the listen ip to the private ip is you are on AWS and change the rpc_ip to 0.0.0.0

also make sure your server has at least 2GB RAM or cassandra will keep crashing!!

to start cassandra on ubuntu try this command:

sudo service cassandra restart

and to check the status do

sudo service cassandra status

Hope this helps

OTHER TIPS

It looks like the Cassandra process is being shut down, presumably from being sent a signal. These lines are what indicate that:

INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,679 CassandraDaemon.java (line 218) Stop listening to thrift clients
INFO [StorageServiceShutdownHook] 2012-07-31 12:44:26,762 MessagingService.java (line 539) Waiting for messaging service to quiesce
INFO [ACCEPT-/10.242.253.141] 2012-07-31 12:44:26,780 MessagingService.java (line 695) MessagingService shutting down server thread.
ERROR [Thrift:40765] 2012-07-31 12:44:26,858 CustomTThreadPoolServer.java (line 204) Error occurred during processing of message.
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top