Cassandra Unbalanced Cluster

Question

Update - Short Version:
The PropertyFileSnitch cassandra-topology.properties property for the first 3 nodes (Rack 1-3) states that only these nodes are in DC1, and the rest are in DC2, specifying the default value default=DC2:r1 . When the cluster was enlarged by adding nodes 4 and 5, the PropertyFileSnitch for these nodes was configured to add them to DC1, as well as to racks 4 and 5, but the snitch of the first 3 nodes remained unchanged and as a result the cluster was in an inconsistent state.

My question is that this cluster can be rebalanced (fixed). Would it be enough if I completely restarted the cluster after fixing cassandra-topology.properties ?
Please advise how I can safely rebalance the cluster.

Longer version:

I am new to Cassandra and I started working on an already built cluster.
I have 5 nodes in one data center on different racks running Cassandra version 3.0.5 with vnodes num_tokens: 256 and a key with replication = {'class': 'NetworkTopologyStrategy', 'DC1': '3'} AND durable_writes = true .
Historically, there were only 3 nodes, and the cluster was enlarged with an additional 2 nodes. I have an automatic repair script that launches nodetool repair with parameters parallelism: parallel, primary range: false, incremental: true, job threads: 1 .

After entering a lot of data, problems arose. When the recovery script starts on node 4 or 5, node 2 reboots: the processor load remains at 100%, the MutationStage queue grows, and the GC pauses to accept for at least 1 s until the Cassandra process finally dies. Typically, the result of the recovery failed with error Stream failed (progress: 0%) .

When I run the nodetool status command on nodes 1, 2, or 3, I get the following output: Datacenter: DC2 Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.0.0.13 10.68 GB 256 0.0% 75e17b8a r1 UN 10.0.0.14 9.43 GB 256 0.0% 21678ddb r1 Datacenter: DC1 Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.0.0.10 16.14 GB 256 100.0% cf9d327f Rack1 UN 10.0.0.11 22.83 GB 256 100.0% e725441e Rack2 UN 10.0.0.12 19.66 GB 256 100.0% 95b5c8e3 Rack3

But when I run the nodetool status command on nodes 4 or 5, I get the following output: Datacenter: DC1 Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.0.0.13 10.68 GB 256 58.9% 75e17b8a Rack4 UN 10.0.0.14 9.43 GB 256 61.1% 21678ddb Rack5 UN 10.0.0.10 16.14 GB 256 60.3% cf9d327f Rack1 UN 10.0.0.11 22.83 GB 256 61.4% e725441e Rack2 UN 10.0.0.12 19.66 GB 256 58.3% 95b5c8e3 Rack3

After further research, it seems that PropertyFileSnitch cassandra-topology.properties did not update on nodes 1, 2, and 3 (which are also seeds for this cluster) after the cluster has been enlarged.

Thanks!

Answer 1

After searching several online resources, I found some possible solutions. I will send them here so that it is available to everyone.

From Practical Cassandra: An Approach to the Developer:

Ring Detection Between Nodes
When the appearance of the ring is different between the nodes, it is never good. There is also an easy way to recover from this condition. The only way to recover is to make a full cluster restart. A rolling restart does not work because the Gossip protocol from bad nodes will inform about new downloads of good nodes from a bad state. A full restart of the cluster and the first launch of good nodes should activate the cluster in good condition.

The same solution can be found in DataStax Documents : The appearance of the ring differs between some nodes p>

I also found a similar question in the Apache Cassandra Community . Response to community user stream:

What happened is that you now have two data center clusters. The way you replicate the information will depend on your Key Settings. As for your process, I don't think it's safe to do it that way. I would start by disabling nodes 4 and 5, so that your cluster goes back to 1 data center with 3 nodes, and then adds them sequentially, making sure that the configuration in Snitch is correct.

Answer 2

I can’t say if what you suggested is enough without resorting to the system, but I have a few comments. Ownership must be distributed among all cluster nodes. This means that the total number of all values ​​on the Dogs tab for all 5 nodes should be 100 if they form one cluster. Having multiple nodes that own a 100% cluster does not look like this. This means that each node acts offline and does not join the cluster.
I see the address 10.40.0.10 in the first printout, and in the second printout 10.0.0.10. Looks like the wrong configuration. In addition, make sure that each node can reach all the other IP addresses of the nodes. I see that 10.0.0.13 refers to 'r1' in the first listing, and in the second to Rack4.
For simplicity and ease of setup, you can configure one data center (e.g., DC1) and one rack (e.g., Rack1) for all 5 nodes, regardless of their physical distribution.