Logstash Grok Filter for uwsgi Logs

Question

I am a new user on the ELK stack. I use UWSGI as my server. I need to parse my wwsgi logs with Grok and then parse them.

Here is the format of my magazines: -

[pid: 7731|app: 0|req: 357299/357299] ClientIP () {26 vars in 511 bytes} [Sun Mar 1 07:47:32 2015] GET /?file_name=123&start=0&end=30&device_id=abcd&verif_id=xyzsghg => generated 28 bytes in 1 msecs (HTTP/1.0 200) 2 headers in 79 bytes (1 switches on core 0) 

I used this link to create my filter, but did not parse most of the information.

The filter generated by the above link,

 %{SYSLOG5424SD} %{IP} () {26 vars in 511 bytes} %{SYSLOG5424SD} GET %{URIPATHPARAM} => generated 28 bytes in 1 msecs (HTTP%{URIPATHPARAM} 200) 2 headers in 79 bytes (1 switches on core 0) 

Here is my logstash-conf file.

 input { stdin { } } filter { grok { match => { "message" => "%{SYSLOG5424SD} %{IP} () {26 vars in 511 bytes} %{SYSLOG5424SD} GET %{URIPATHPARAM} => generated 28 bytes in 1 msecs (HTTP%{URIPATHPARAM} 200) 2 headers in 79 bytes (1 switches on core 0)" } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } } output { stdout { codec => rubydebug } } 

After running logstash with this conf file, I get the error message: -

 { "message" => "[pid: 7731|app: 0|req: 357299/357299] ClientIP () {26 vars in 511 bytes} [Sun Mar 1 07:47:32 2015] GET /?file_name=123&start=0&end=30&device_id=abcd&verif_id=xyzsghg => generated 28 bytes in 1 msecs (HTTP/1.0 200) 2 headers in 79 bytes (1 switches on core 0)", "@version" => "1", "@timestamp" => "2015-03-01T07:57:02.291Z", "host" => "cube26-Inspiron-3542", "tags" => [ [0] "_grokparsefailure" ] } 

The date was formatted correctly. How to extract other information from my logs, for example, my query parameters(filename, start,end, deviceid etc) and ClientIP , Response code , etc.

In addition, is there a built-in UWSGI log analyzer that can be used, for example, one that was created for apache and syslog?

EDIT

I wrote this myself, but it throws the same error:

 %{SYSLOG5424SD} %{IP:client_ip} () {%{NUMBER:vars} vars in %{NUMBER:bytes} bytes} %{SYSLOGTIMESTAMP:date} %{WORD:method} %{URIPATHPARAM:request} => generated %{NUMBER:generated_bytes} bytes in {NUMBER:secs} msecs (HTTP/1.0 %{NUMBER:response_code}) %{NUMBER:headers} headers in %{NUMBER:header_bytes} (1 switches on core 0) 

EDIT 2

Finally, I was able to crack it. The GROK filter for the specified log will be:

 \[pid: %{NUMBER:pid}\|app: %{NUMBER:app}\|req: %{NUMBER:req_num1}/%{NUMBER:req_num2}\] %{IP:client_ip} \(\) \{%{NUMBER:vars} vars in %{NUMBER:bytes} bytes\} %{SYSLOG5424SD} %{WORD:method} /\?file_name\=%{NUMBER:file_name}\&start\=%{NUMBER:start}\&end\=%{NUMBER:end} \=\> generated %{NUMBER:generated_bytes} bytes in %{NUMBER:secs} msecs \(HTTP/1.0 %{NUMBER:response_code}\) %{NUMBER:headers} headers in %{NUMBER:header_bytes} 

But my questions still remain:

• Is there any default uwsgi filter file in grop ?? **

• I apply different matches for different request parameters. Is there anything in grok that selects various query parameters?

Answer 1

I found a solution for retrieving query parameters: -

Here is my final configuration: -

For log line

 [pid: 7731|app: 0|req: 426435/426435] clientIP () {28 vars in 594 bytes} [Mon Mar 2 06:43:08 2015] GET /?file_name=wqvqwv&start=0&end=30&device_id=asdvqw&verif_id=qwevqwr&lang=English&country=in => generated 11018 bytes in 25 msecs (HTTP/1.0 200) 2 headers in 82 bytes (1 switches on core 0) 

configuration

 input { stdin { } } filter { grok { match => { "message" => "\[pid: %{NUMBER}\|app: %{NUMBER}\|req: %{NUMBER}/%{NUMBER}\] %{IP} \(\) \{%{NUMBER} vars in %{NUMBER} bytes\} %{SYSLOG5424SD:DATE} %{WORD} %{URIPATHPARAM} \=\> generated %{NUMBER} bytes in %{NUMBER} msecs \(HTTP/1.0 %{NUMBER}\) %{NUMBER} headers in %{NUMBER}" } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } kv { field_split => "&? " include_keys => [ "file_name", "device_id", "lang", "country"] } } output { stdout { codec => rubydebug } elasticsearch { host => localhost } } 

Answer 2

I found that your solution does not support HTTP / 1.1. I fixed it and added the variable name. Link

Here is my grok config:

 grok { match => { "message" => "\[pid: %{NUMBER:pid}\|app: %{NUMBER:id}\|req: %{NUMBER:currentReq}/%{NUMBER:totalReq}\] %{IP:remoteAddr} \(%{WORD:remoteUser}?\) \{%{NUMBER:CGIVar} vars in %{NUMBER:CGISize} bytes\} %{SYSLOG5424SD:timestamp} %{WORD:method} %{URIPATHPARAM:uri} \=\> generated %{NUMBER:resSize} bytes in %{NUMBER:resTime} msecs \(HTTP/%{NUMBER:httpVer} %{NUMBER:status}\) %{NUMBER:headers} headers in %{NUMBER:headersSize} bytes %{GREEDYDATA:coreInfo}" } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] }