Why do I need a broker for my production ELK stack + machine specifications?

Question

I recently got into the ELK stack Ubuntu test block to test the functionality and was very pleased with it. My use case for production will include at least 100 GB of magazines per day. I want to be as scalable as possible, as this 100 GB / day can grow rapidly, as we had more sources of magazines.

I read several articles about ELK production, including the fantastic Logz.io ELK Deployment . Although I have a general idea of ​​what I need to do, I’m not sure about some basic concepts, about how many machines I need for such a large amount of data and whether I need a broker like Redis in my architecture.

What is the point of a broker like Radish? In my test instance, I have several log sources sending logs via TCP, syslog and logstash forwarder to my Logstash directly on my ELK server (which also has Elasticsearch, Nginx, and Kibana configured using SSL).

To maintain a high level of availability, a modern production cluster, which machines + specifications do I need for at least 100 GB of data per day, is it likely to scale to 150 GB or more in the future? I plan to use my own servers. From what I explored, the starting point should be something like (if you enable Redis):

• 2/3 of the server with an instance of Redis + Logstash (indexer) for each server. For specs, I think 32 GB of RAM, 500 GB fast I / O, possibly SSD, 8 cores (i7)
• 3 servers for Elasticsearch (this is the one I'm very unsure about). I know that I need at least 3 master nodes and 2 data nodes, so 2 servers will have 1 master-1 data each - these will be 64-bit RAM, 20 TB, 8 cores. The other remaining node wizard may be on a machine with a low specification, as it does not process data.
• 2 servers for Nginx / Kibana - these should be low-specialized machines, as this is just a web server and user interface. Is load balancing required?

EDIT: Planning log retention for 60 days.

Answer 1

As for Redis, it acts as a buffer in case the logstash and / or elasticsearch are slow or slow. If you use a full logstast or logstast forwarder as a shipper, it detects that logstash is unavailable and stops sending logs (remembering where it left off, at least for a while).

So, in a clean logstash / logstash-forwarder environment, I see no reason to use a broker like redis.

When it becomes important, for sources that do not care about start status and do not buffer them by side. syslog, snmptrap and others fall into this category. Since your sources include syslog, I would drop brokers in your setup.

Redis is an RAM-intensive application, and the amount of memory you have will determine how much time you can withstand due to a malfunction in the log. On a 32 GB server (along with logstash), how much memory would you give yo redis? How big is your average document size? How many documents will it take to fill memory? How long does it take to create many documents? In my experience, redis fails when memory is full, but it can only be me.

Logstash is a CPU intensive process as all filters are executed.

Regarding elasticsearch cluster size, @magnus has already pointed you to some information that might help. Starting with 64 gigabyte machines, fine, and then scale horizontally as needed.

You should have two client (non-data) nodes that are used as the access point for the inserts (efficiently sending requests for the correct node data) and search (the “reduce” phase processing with the data returned from the data nodes). Two of these failover configurations will be a good start.

Two kibana machines will provide you redundancy. Including them in a failover configuration is also good. In my opinion, nginx was used more with kibana3. I do not know if people use it with kibana4 or switched to a "shield".

Hope this helps.