All geek questions in one place

Cassandra node cannot start after hard drive failure

I have a 5 node Cassandra 2.0.7 cluster, each node has 4 hard drives. Recently, one of these hard drives on host3 failed and was replaced by a new shiny blank drive. After replacing cassandra on this node, it was not possible to start with this exception:

INFO [main] 2014-06-02 12:45:17,232 ColumnFamilyStore.java (line 254) Initializing system.paxos INFO [main] 2014-06-02 12:45:17,236 ColumnFamilyStore.java (line 254) Initializing system.schema_columns INFO [SSTableBatchOpen:1] 2014-06-02 12:45:17,237 SSTableReader.java (line 223) Opening /mnt/disk2/cassandra/system/schema_columns/system-schema_columns-jb-310 (25418 bytes) INFO [main] 2014-06-02 12:45:17,241 ColumnFamilyStore.java (line 254) Initializing system.IndexInfo INFO [main] 2014-06-02 12:45:17,245 ColumnFamilyStore.java (line 254) Initializing system.peers INFO [SSTableBatchOpen:1] 2014-06-02 12:45:17,246 SSTableReader.java (line 223) Opening /mnt/disk3/cassandra/system/peers/system-peers-jb-25 (20411 bytes) INFO [main] 2014-06-02 12:45:17,253 ColumnFamilyStore.java (line 254) Initializing system.local INFO [SSTableBatchOpen:1] 2014-06-02 12:45:17,254 SSTableReader.java (line 223) Opening /mnt/disk3/cassandra/system/local/system-local-jb-35 (80 bytes) INFO [SSTableBatchOpen:2] 2014-06-02 12:45:17,254 SSTableReader.java (line 223) Opening /mnt/disk3/cassandra/system/local/system-local-jb-34 (80 bytes) ERROR [main] 2014-06-02 12:45:17,361 CassandraDaemon.java (line 237) Fatal exception during initialization org.apache.cassandra.exceptions.ConfigurationException: Found system keyspace files, but they couldn't be loaded! at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:532) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:233) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:462) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:552)

Due to the inability to start cassandra node, I cannot use nodetool repair .

The only way to restore the node is to delete all the data and load it from almost bare metal. Is there a shorter recovery method in a typical hard drive failure scenario?

+6
source share
2 answers

The problem with these steps has been fixed:

  • physically deleted files associated with system keyspace: cassandra managed to run and recreate it, but without metadata about other key spaces.

  • launched nodetool resetlocalschema , which synchronized the nodetool resetlocalschema scheme from other nodes.

+16
source

Another solution I found was to simply delete the data directory in the Cassandra parent directory. After starting Cassandra again, it should recreate your namespace.

0
source

Source: https://habr.com/ru/post/970167/

More articles:


All Articles