All geek questions in one place

Java - Best practice for logging in a multi-node environment

in my company, I manage a large application (> 100 thousand users) in an infrastructure with several nodes consisting of 3 (but potentially more) application servers. Each application server has 5 different log files on which almost all information about HTTP / REST or SOAP requests and responses to other (external) subsystems are logged. I use the Apache Http client to handle REST streams generated by wsimport clients for SOAP requests and Logback as a logging technology.

At the moment, when I am asked to debug something, the most difficult and time-consuming task I have to perform is to identify the node on which I have to debug. After that, I really have to wipe tons of lines to find out what happened. Honestly, I find this very boring and outdated, as well as complicated.

To make my life easier and make my magazines more interesting, I looked in the past days on the elasticsearch stack (elasticsearch, logstash, kibana) and played with my docker images. I find them very interesting and I would like to introduce them in my application, but before doing this, I would like to know if there are any best practices / templates to do something like this.

These are my doubts:

  • Is there a best practice for registering the Http / s REST and SOAP / response protocols (I need to see everything: url, headers, path, body, cookies, ...) in a format that can be easily analyzed using logstash / elasticsearch?
  • Given my infrastructure, should I use the elasticsearch application for my login implementation or use Logstash as a log handler (I assume there is one for each application server)?
  • Are there any valid alternatives to the logback and elasticsearch methods to fulfill my requirements?

I do not expect a simple and easy answer. I would like to read about various experiences in order to make the choice that best suits my decision.

Thank!

+4
3

:) ELK . , , :

  • , ElasticSearch, ( , , ):

    • ,
    • ,
    • - , ,

  • ()

  • Java, ( ), . Logstash ,

  • ( , 3 ), . , .

Elastic Search .

TTL . , , , .

HTTP. , , "" . , ( , :)). , URL, , http ( , ) , .

appender vs logstash (files). . : logstash, . , , CPU ( grok logstash). - ( java).

, . logback Elasticsearch appender, , :

  • ( - )
  • ( , ES - . ).
  • , / , , , appender ES, . , 1000, 1000 , , FIFO.

, : , - . , , ( ). im-memory, ? ElasticSearch post mortum? , .

, , , .

appender logstash, , Apache Flume . appender, flume agent . Flume , , api ..

, logstash, .

, , , ​​ :

  • ElasticSearch. ( logstash Redis RabbitMQ). - ( out out out of the box).

ElasticSearch - ( Redis/get RabbitMQ). .

+3

[...] format [/protocol]?

- , . , . syslog, RFC5424. , JSON .

elasticsearch?

logstash ElasticSearch.

  • Logstash , , logstash, , ElasticSearch.

  • : - Logstash - .

    • , ElasticSearch.

  • Logstash , ( , )

  • Logstash . : PagerDuty Slack, .
  • Logstash (, IP-)

. , , : , Logstash , ( ). , ElasticSearch, ElasticSearch, . , Logstash .

, ElasticSearch. . , , :

  • ,
  • ( )
  • ,

- , , (, syslog appender).

logback elasticsearch ?

, logstash ( " ELK"?) , .

logback... log4j2. , . , logback . log4j2 ( JSON. , ). - syslog RFC5424.

Java (.. SLF4J), . .

+1

, , (, Logstash/Flume).

https://github.com/internetitem/logback-elasticsearch-appender , , https://github.com/logstash/logstash-logback-encoder

, logstash-logback-encoder https://logback.qos.ch/apidocs/ch/qos/logback/classic/spi/ILoggingEvent.html

logback-elasticsearch-appender FS, ES- ( ).

, Logstash , https://www.elastic.co/guide/en/logstash/current/persistent-queues.html

Logstash ( → ) .

, - Redis, RabbitMQ Kafka. , ES- , Logstash ( ES Elasic.io).

, Logstash Ruby ! . 10000 req/s ( , ).

. , . ))

:

  • Splunk <http://www.splunk.com/en_us/products/splunk-light.html>
  • Scalyr <https://www.scalyr.com/pricing>
  • Graylog <https://www.graylog.org/support-packages/>
  • Loggly <https://www.loggly.com/product/>
  • Motadata <https://www.motadata.com/elk-stack-alternative/>

.

, appender : https://logz.io/blog/lessons-learned-writing-new-logback-appender/

:

0
source

Source: https://habr.com/ru/post/1675170/

More articles:


All Articles