Cassandra with DR DC has unexpected uneven data distribution

Question

We are trying to provision a Cassandra cluster to use as a KV storage.

We are deploying a 3-node production cluster on our production DC, but we would also want to have a single node as a DR copy on our disaster recovery DC.

Using PropertyFileSnitch we have

 10.1.1.1=DC1:R1
 10.1.1.2=DC1:R1
 10.1.1.3=DC1:R1
 10.2.1.1=DC2:R1

We plan on using a keyspace with the following definition:

CREATE KEYSPACE "cassandraKV"
  WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'DC1' : 2, 'DC2' : 1};

In order to achieve the following: 2 replicas distributed among 3 nodes in DC1 (66% of total data per node) while still allowing a single node to go down without any data loss. 1 replica in DC2 (100% of total data per node)

We see the ownership distributed at 25% per node, while we would expect 33% on each node in DC1 and 100% in DC2.

Is this above configuration correct?

Thanks

Answer 1

My guess is that you ran nodetool Status without specifying the keyspace. This will end up just showing you the general distribution of the tokens in your cluster which will not be representative of your "cassandraKV" keyspace.

On running nodetool status "cassandraKV" you should see

Datacenter: DC1

10.1.1.1: 66%
10.1.1.2: 66%
10.1.1.3: 66%

Datacenter: DC2

10.2.1.1: 100%

You should see 66% for the DC1 nodes because each node holds 1 copy of it's primary range (33%) and one copy of whatever it is replicating (33%)

While DC2 has 100% of all the data you are currently holding.