Cassandra's stress test fails after a large number of inserts

Question

I am trying to insert 1 billion records in Cassandra using a stress test, and it fails after several million inserts with the following error:

Operation [641412926] repeated 10 times - error input key 0641412926 ((UnavailableException))

Operation [641412995] repeated 10 times - error input key 0641412995 ((UnavailableException))

Operation [641413235] repeated 10 times - error input key 0641413235 ((UnavailableException))

Operation [641413164] repeated 10 times - error input key 0641413164 ((UnavailableException))

I observed this problem during every stress test. Sometimes any of the nodes in the cluster goes down. Is this a known issue? Any specific reason why this is happening? I am using Cassandra 1.2.3 on a cluster of 8 machines.

Thanks, VS

Answer 1

UnavailableException means that the node you contacted cannot find enough replicas in the cluster for the key requested to execute the request. If you have nodes that rise and fall during the stress test, you probably need more options to handle the load that you are using against the cluster.

Why is this happening? You are probably not capable enough. If you do not have enough disk space, you should evaluate the load on your CPU and your IO to try and figure out what is happening. When using Cassandra, it is important to distinguish between peak load and constant load. While Cassandra can handle ephemeral peaks, it is possible to load more load on a node than it can handle in the long run. This means that if your peak lasts five minutes, you will probably be fine. If your peak lasts for several days, you should add capacity because your cluster will eventually lag behind.

Answer 2

The first thing to check is to insert the node into up and cassandra. Assuming this is so, then you can be an overwhelming cassandra. In general, applications running in the JVM cannot recover when the JVM garbage collection process crashes. This may be the cause of the error you are causing, and therefore your Cassandra node is not being restored. To verify that this is the case, enable more detailed GC protocols and / or consult existing JVM GC log messages in system.log.