Solr problem: ClusterState says we are the leader, but locally we don’t think so

Question

So today we are faced with the problem of anxiety. After rebooting the entire cluster, one of the fragments ceases to be able to index / store documents. We did not know about the problem until we started indexing (the request to the server looks great). Error:

2014-05-19 18:36:20,707 ERROR o.a.s.u.p.DistributedUpdateProcessor [qtp406017988-19] ClusterState says we are the leader, but locally we don't think so
2014-05-19 18:36:20,709 ERROR o.a.s.c.SolrException [qtp406017988-19] org.apache.solr.common.SolrException: ClusterState says we are the leader     (http://x.x.x.x:7070/solr/shard3_replica1), but locally we don't think so. Request came from null
  at org.apache.solr.update.processor.DistributedUpdateProcessor.doDefensiveChecks(DistributedUpdateProcessor.java:503)
  at org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:267)
  at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:550)
  at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:126)
  at org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:101)
  at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:65)
  at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
  at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916)

We launch Solr 4.7 in cluster mode (5 shards) on the pier. Each shard runs on a different host with one zookeeper server.

I checked the zookeeper log and I can't see anything there.

The only difference is that in the / overseer _election / election folder, I see that this particular server is repeated 3 times, and the other server is only mentioned twice.

  45654861x41276x432-x.x.x.x:7070_solr-n_00000003xx
  74030267x31685x368-x.x.x.x:7070_solr-n_00000003xx
  74030267x31685x369-x.x.x.x:7070_solr-n_00000003xx

Not sure if this is relevant. (Could this be so?) Do you know what else you can check?

Answer 1

! , , 2 , - , .

Java- .

Answer 2

.

1

zookeeper Zookeeper ephemeral node /overseer_elect/election. node , .

node . : https://issues.apache.org/jira/browse/ZOOKEEPER-2355

/overseer/queue, , .

, Zookeeper node node.

Still seeing conflicting information about the leader of shard shard1 for collection <name> after 30 seconds Solr, .

2

: systemd. , Type=forking PIDFile, systemd.

systemd PID , , , , - 2 . ( ), , , , - solr , .

Solr , OP.

, , zookeeper /collections/<name>/leaders/shard1/leader, , zk node , :

{ "": "-name_shard1_replica1", "Core_node_name": "core_node7", "base_url": " http://10.10.10.21:8983/solr", "Node_name": "10.10.10.21:8983_solr" }

node solr, .

Solr: HttpSolrCall null:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json

, solr ( java, , ) solr.