How to prevent Cassandra logs from filling up disk space

Question

I am running two Datastax AMI cluster nodes on AWS. Cassandra yesterday began to abandon ties with everything. System logs showed nothing. After many attempts, I found that the commit logs filled up all of the disk space on the dedicated mount, and this seemed to cause a connection failure (deleted some of the commit logs, restarted, and was able to connect).

I'm on DataStax AMI 2.5.1 and Cassandra 2.1.7

If I decide to erase and restart everything from scratch, how can I make sure that this does not happen again?

Answer 1

You can try omitting the commitlog_total_space_in_mb parameter in cassandra.yaml . By default, 8192 MB is used for 64-bit systems (it should be commented out in your .yaml file ... you will not have to comment on it when setting up). It is generally recommended that you plan this when determining the size of your disk (s).

This can be verified by running du in your commitlog directory:

 $ du -d 1 -h ./commitlog 8.1G ./commitlog 

Although a smaller logarithmic commit space will cause more frequent shifts (increased disk I / O), so you will need to keep track of this.

Answer 2

In addition to reducing the size of commitlog, as suggested by BryceAtNetwork23, the right solution to ensure that it repeats will not lead to monitoring the installation of the disk, so that you are warned about filling it up and have time to act / increase the disk size.

When you see how you are using DataStax, you can set an alert for this in OpsCenter. I have not used this in the cloud, but I suppose it will work. Alerts can be set by clicking "Alerts" in the top banner → "Manage Alerts → Add Alert". Configure mounts for viewing and thresholds for inclusion.

Or, I'm sure there are better tools for monitoring disk space.