So, I have this problem with Kibana, that I do not know how to solve it. Let's start with a description of the environment:
- 12 Elasticsearch (version 2.1.2). These are various AWS machines with Linux / CentOS installed, using r3 / r4 / i3 instance types distributed across three AZs. Each car has:
- 800 GiB storage (EBS and SSD)
- 30 GiB RAM
- Kibana (version 4.3.3)
- elasticsearch.requestTimeout and elasticsearch.shardTimeout are set to 1500000, i.e. 1500 seconds, i.e. 25 minutes.
- We create 26 indexes per day, keep them for 12 days, most of them have 12 fragments and 1 replica.
- AWS ELB, which distributes requests (non-sticky) to all 12 nodes.
- The idle timeout for ELB is 900 seconds, i.e. 15 minutes.
Now, when launching more complex queries, Kibana, when accessed via ELB, displays a “gateway timeout” error message after two to three minutes. The network console in Chrome shows that two queries are launched, the first defining indexes for the requested time period, the second for the result data. The first request takes 20 to 50 seconds and succeeds, the second request always ends 504 (gateway timeout) after exactly two minutes.
When starting a request generated by Kibana directly against one of the Elasticsearch nodes, the request takes about 7 minutes and succeeds.
When executing a request without ELB directly against one of the Kibana instances (using curl), it also fails after almost exactly two minutes. Using Wireshark, I see that Kibana just closes the HTTP connection after 120 seconds.
-? ?