All geek questions in one place

Cassandra nodes cannot communicate with each other, call ReadTimeout

This is the version of Datastax Cassandra (dse): 4.8.5-1
This corresponds to (I believe) Cassandra: 2.1.x

When requesting from our application, the following errors occur:

ReadTimeout: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'data_retrieved': False, 'required_responses': 1, 'consistency': 1}

Digging into it more; an example request (starting using cqlsh locally on each node) returns to 3 of the nodes in the ring, but ReadTimeout is not executed with the rest. It seems that only nodes containing replicas return with the answer, and the rest do not know how to find them at all.

Is there some kind of configuration or known issue that I have to solve in order to fix this problem?

When other nodes fail, I see this error in the logs:

 ERROR [MessagingService-Outgoing-/10.0.10.14] 2016-04-25 20:46:46,818 CassandraDaemon.java:229 - Exception in thread Thread[MessagingService-Outgoing-/10.0.10.14,5, main] java.lang.AssertionError: 371205 at org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:290) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:393) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:382) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:271) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:259) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:503) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:490) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromReadCommand.java:168) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:330) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:282) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]

Nodetool Status Output

 Datacenter: primary ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 10.0.10.224 557.95 GB 1 ? d1b984b0-50d4-4faa-b349-08bc0cf36447 RAC1 UN 10.0.10.225 740.11 GB 1 ? 16ab3c8c-476e-46c2-837c-6dbb89b7d40d RAC1 UN 10.0.10.12 748.23 GB 1 ? 4127f0d7-6bd0-4dc8-b6a0-3b261e55b44e RAC1 UN 10.0.10.45 629.27 GB 1 ? f4499c5d-f892-43b8-97f3-dcce5be51fb8 RAC2 UN 10.0.10.13 592.57 GB 1 ? 41b58044-942d-4e77-a8de-95495b88a073 RAC1 UN 10.0.10.14 616.45 GB 1 ? d2b568fb-13e1-4ff7-a247-3751a8ca49cf RAC1 UN 10.0.10.15 623.23 GB 1 ? fb10e521-8359-409b-bfd8-b27829157a80 RAC1 UN 10.0.10.21 538.56 GB 1 ? 72288b4c-bd1d-4398-9d95-5af312c2f904 RAC2 UN 10.0.10.25 616.63 GB 1 ? 4a8f04ff-a198-44d1-baf4-72cc430cd8a9 RAC2 UN 10.0.10.218 562.98 GB 1 ? c00c375d-90bb-48c5-a8d0-7102a13db468 RAC2 UN 10.0.10.219 632.58 GB 1 ? 1e2ea144-35bd-412b-89b5-41544a347a75 RAC2 UN 10.0.10.220 746.85 GB 1 ? d40f59c1-430a-4d96-9d7e-1e846b8eb1fc RAC2 UN 10.0.10.221 575.89 GB 1 ? 7e407d6b-2bd5-43b4-9116-96ee72a926b2 RAC2 UN 10.0.10.222 639.98 GB 1 ? bfd04ab8-7679-4474-8d47-984950bdd2c7 RAC1 UN 10.0.10.223 652.58 GB 1 ? 6366cd3e-7910-40bb-8a12-926c53adf95b RAC1

The code for this statement is here:

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.1.1/org/apache/cassandra/utils/ByteBufferUtil.java?av=f#290

  • There is no obvious schema mismatch when viewing system.local or system.peers .
  • nodetool describecluster returns UNREACHABLE from some nodes
+5
source share
2 answers

You probably fall into the 64K maximum key size limit, http://wiki.apache.org/cassandra/FAQ#max_key_size

Look at your application code, maybe someone will send cassandra 371205 bytes as the primary key, maybe someone is trying to crack your application, I don’t know, because the 370k extremely unlikely data as the primary key is reasonable, limit it in your application code

I do not know if there is an error error or a fix or workaround.

+4
source

1. Check replication rate. 2. Check with a different level of consistency. 3. In most cases, you may be mistaken in the fact that you give only one token per node. If you have 256 tokens per node, then cassandra distributes the load to each node.

0
source

Source: https://habr.com/ru/post/1247896/

More articles:


All Articles