Logstash converts date to actual joda time (@timestamp)

Question

Hope someone can help me!

I have a question about logstash. I get the following date with success: June 26/2013: 14: 00: 26 +0200

Next, I want this date to be used as @timestamp events. As you know, logstash automatically adds a timestamp.

Replacing the timestamp that the logstash adds can be done using a date filter. I added the following date filter: match => ["date", "dd / MMM / YYYY: HH: mm: ss Z"]

But for some reason this does not work. When I check this, I see that logstash just adds its own timestamp.

code:

grok { type => "log-date" pattern => "%{HTTPDATE:date}" } date{ type => "log-date" match => [ "date", "dd/MMM/YYYY:HH:mm:ss Z"] } 

I need to do this, so I can add events to elasticsearch.

Thanks in advance!

Answer 1

I used the following approach:

 # strip the timestamp and force event timestamp to be the same. # the original string is saved in field %{log_timestamp}. # the original logstash input timestamp is saved in field %{event_timestamp}. grok { patterns_dir => "./patterns" match => [ "message", "%{IRODS_TIMESTAMP:log_timestamp}" ] add_tag => "got_syslog_timestamp" add_field => [ "event_timestamp", "%{@timestamp}" ] } date { match => [ "log_timestamp", "MMM dd HH:mm:ss" ] } mutate { replace => [ "@timestamp", "%{log_timestamp}" ] } 

Now my problem is that even if @timestamp is replaced, I would like to first convert it to an ISO8601 compatible format so that other programs do not have problems interpreting it, like the timestamp present in event_timestamp:

  "@timestamp" => "Mar 5 14:38:40", "@version" => "1", "type" => "irods.relog", "host" => "ids-dev", "path" => "/root/logstash/reLog.2013.03.01", "pid" => "5229", "level" => "NOTICE", "log_timestamp" => "Mar 5 14:38:40", "event_timestamp" => "2013-09-17 12:20:28 UTC", "tags" => [ [0] "got_syslog_timestamp" ] 

You can easily convert it, since you have the information of the year ... In my case, I would have to parse it from the attribute "path" (filename) ... but still, it seems that there is no convert_to_iso8901 => @timestamp directive.

Hope this helps with your problem anyway! :)

Answer 2

The above answer is just work !, try adding locale => "en" to your code.
If it is not added, the date days of the week and monthly names will be processed by the language of the platform’s language platform by default (Spanish, French or any other), and therefore it does not work (since your journal is in English).

 date{ type => "log-date" match => [ "date", "dd/MMM/YYYY:HH:mm:ss Z"] locale => "en" }