ELK processes multi-line logs from multiple docker images

Question

I am running ELK (Elasticsearch, Logstash, Kibana) in a cluster where docker containers are running. These containers send logs to Logstash through the GELF endpoint.

docker run  --log-driver=gelf --log-opt gelf-address=udp://somehost:12201 -ti    my-app:latest 

And then I process the logs in Logstash. Here I want to collapse multiline messages and combine them into one event (Java exception in my case). My configuration:

input {
    gelf {} 
}
filter{
    multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
      source => "short_message"
      }
}
output {
    stdout { codec => rubydebug }
}

It works great when I process the logs from one docker container, but for two or more it does not work, because it collapses the messages of both (or more) log streams.

I would expect that setting multilevel input would solve the problem

input {
    gelf {
      filter{
         multiline {
            pattern => "^%{TIMESTAMP_ISO8601}"
            negate => true
            what => "previous"
         }
     }
}

but multi-line operation does not work with this setting (it seems due to an error ). Any suggestions? Thanks.

: Docker 1.9.1, Logstash 2.1

Answer 1

, 'stream_identity' .

, , . , , , , , tcp.

https://www.elastic.co/guide/en/logstash/current/plugins-filters-multiline.html#plugins-filters-multiline-stream_identity

Gelf, host container_id :

filter {

  multiline {
    pattern => "^%{TIMESTAMP_ISO8601}"
    negate => true
    what => "previous"
    source => "short_message"
    stream_identity => "%{host}.%{container_id}"
  }
}