Logstash: Received an event that has a different character encoding

Question

when using logstash, I see an error similar to this:

Received an event that has a different character encoding than you configured. {:text=>"2014-06-22T11:49:57.832631+02:00 10.17.22.37 date=2014-06-22 time=11:49:55 device_id=LM150D9L23000422 log_id=0312318759 type=statistics pri=information session_id=\\\"s617nnE2019973-s617nnE3019973\\\" client_name=\\\"[<IP address>]\\\" dst_ip=\\\"<ip address>\\\" from=\\\" [e-mail] \\\" to=\\\" [e-mail] \\\" polid=\\\"0:1:1\\\" domain=\\\"machin.fr\\\" subject=\\\"\\xF0\\xCC\\xC1\\xD4\\xC9 \\xD4\\xCF\\xCC\\xD8\\xCB\\xCF \\xDA\\xC1 \\xD0\\xD2\\xCF\\xC4\\xC1\\xD6\\xC9!\\\" mailer=\\\"mta\\\" resolved=\\\"OK\\\" direction=\\\"in\\\" virus=\\\"\\\" disposition=\\\"Quarantine\\\" classifier=\\\"FortiGuard AntiSpam\\\" message_length=\\\"1024\\\"", :expected_charset=>"UTF-8", :level=>:warn} 

my logstash.conf:

  input { file{ path => "/var/log/fortimail.log" } 

}

  filter { grok { # grok-parsing for logs } } output { elasticsearch { host => "10.0.10.62" embedded => true cluster => "Mastertest" node_name => "MasterNode" protocol => "http" } } 

I don’t know which codec should be used for the correct event format? It is in the subject field.

Answer 1

This is due to the fact that the default encoding is UTF-8, and the incoming message contains a character not in the UTF-8 set

To fix this set, the encoding in the input section using the codec and the correct encoding. for instance

 file { path => "var/log/http/access_log" type => apache_access_log codec => plain { charset => "ISO-8859-1" } stat_interval => 60 } 

http://logstash.net/docs/1.3.3/codecs/plain

Answer 2

If you got the logs from an external server, try using:

 input { udp { port => yourListenPort type => inputType codec => plain { charset => "ISO-8859-1" } } } 

I got the same error and I used this, it works!