Cassandra Exception Cache scheme X does not match current version of scheme Y

Question

We had DataStax Cassandra Community Server 3.0.2 installed on our mailboxes, several keys were created, and we transferred some data from our old database to Kassandra (Aroudnd 10 GB of data). Although this happened, we created materialized views in these tables. A few minutes (not sure exactly how long) after the creation, we received our services (deployed on ~ 2x3 nodes in 2 data center architectures) that were accidentally stopped with absolutely no log information about why it stopped. After that we tried to start our service, we got the following error in our logs

 java.lang.RuntimeException: Cache schema version a2c390a1-f2cd-3d59-8b5c-a0a1d794d0fe does not match current schema version 8d1347d7-7729-3698-8537-4b91ae9ce7dd at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:198) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:157) [apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:153) [apache-cassandra-3.0.2.jar:3.0.2] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_66] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] 

We tried to look into the code in which this expression could occur. This gave us some information about what causes it - apparently, the service is trying to match the current version of the scheme with the one that is present in the cache (I would assume it either before the service stops / creating materialized views) We are very sure that we did not schema changes in key space other than creating materialized representations.

So, what makes me wonder, has no one reported such a problem in the past? Therefore, we are not sure what will go wrong on our side. We hired nodetool cleanup on our nodes without help. We do not want to lose any data. We are trying to restore, so we deleted the $CASSANDRA_HOME\data\saved_caches and restarted the service. When you start the server, it plays back transactional records and takes a lot of time. I suppose this could be due to the amount of data that we have or the machines have completely disappeared and need to be replaced?

Any pointers related to the problem can be really helpful! Thank you in advance.

If this can help, we run DataStax Cassandra Server on Windows machines and run Cassandra as a Windows service.

Answer 1

Removing the commitlog directory and restarting it did this for me.

 rm -rf /path/to/cassandra/data/commitlog 

Late, but I hope this helps anyone who is stuck.

Answer 2

Cache Schema Version ...

Clear the saved_caches directory not the commit_log directory. Removing commit_logs is dangerous. Commit logs can save a day and prevent data loss if your nodes unexpectedly crash and your write data has not been reset.