ElasticSearch introduces read-only mode, node cannot be changed

Question

Something happened at night in my ES cluster (consisting of 5 data nodes, 3 main nodes).

Honestly, I have no idea what happened, but all the indexes and data were deleted, and the cluster went into read-only mode, maybe hacked?

When I try to start Kiban, I get the following:

I tried restarting the kibana - it restarted, nothing has changed. I tried restarting Elastic - it restarted (all nodes), nothing has changed.

Then I looked at the cluster settings, and here is what I got:

{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      },
      "blocks": {
        "read_only": "true"
      }
    }
  },
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      }
    }
  }
}

I tried to cancel reading only as follows:

PUT _cluster/settings
{
  "persistent": {
    "blocks.read_only": false
  }
}

You were unlucky:

{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
  },
  "status": 403
}

Any ideas?

: , - ? ? , .

Answer 1

:

PUT /_cluster/settings
{
  "persistent" : {
    "cluster.blocks.read_only" : false
  }
}