Java/Hector cassandra-client different performance on different OS

Question

we are using Cassandra on 5 nodes replication 3 for over a year now.

Current Version 1.2.12

We wrote a program deleting records in Cassandra via hector/Java.

On the old OS (OpenSuSE) we had a write performance of about 2500 ops/sec. Now we are upgrading the nodes to Ubuntu 12.04 LTS. (Cassandra was not touched, has it own raid-system). Due to that we move the Deleting-Programm to an other node with new OS. On this node the write performance is about 70 ops/sec, which is not acceptable.

We are using the same JRE and same Programm-Code on both machines.

/etc/security/limits.conf 

are the same.

Profiling the java code on the slow machines say bottleneck is: org.apache.thrift.transport.TIOStreamTransport.read()

Does anywone know if there are any OS-setting wich slow down the cassandra operations? Thanks for your help.

---

Update: (day 3)

• Noticed that on machines with kernel 3.5 and lower the performance is OK ... keep searching
• machines with kernel 3.8 have bad performance

---

Update: (day 4)

The "OS/kernel-suspicion" was wrong / now I have 2 ubuntu machines with same kernel Version / one is fast, one is slow

... no firewall ... checking network :P

---

Update: (day 5)

I did some cross-test (connecting each node with each other by AllOneConsistencyLevelPolicy) I've got more problems with cassandra-clients using Kernel 3.8+ contacting Cassandra-servers kernel <=3.5.

• 3.8 to 3.8 some slow, some fast
• 3.8 to <3.5 some slow, some fast
• <3.5 to any other - always fast

I've no idea where to search next... brainstorming welcome ;)

---

Still slow... Just Connections to localhost are (always) fast. Having this phenomena on two different clusters in two different datacenters (also two different providers). So it's most likely it has nothing to do with the network infrastructure.

Cassandras has also different Versions in the second datacenter (2.0.6)

So I still think it's some Kernel-Issue ... no one else with this problem?

---

Found it ... we are running cassandra as a non-root-user. There is an extra "switch" in ubuntu which limits the number of open files. It was following thread.

https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user

Cite: [...]

You also need to edit /etc/pam.d/common-session* and add the following line to the end:

session required pam_limits.so

[...]

did the trick.

Answer 1

Make sure your resource limits and raid configuration are the same on both. Readahead is a particularly commonly misconfigured setting; see "setra" on this page: http://www.datastax.com/documentation/cassandra/2.0/webhelp/cassandra/install/installRecommendSettings.html

Special for Ubuntu with Cassandra running as non-root user:

You also need to edit /etc/pam.d/common-session* and add the following line to the end:

session required pam_limits.so

cp. https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user