Logstash: how to include input line number

Question

I am trying to create a way to navigate my log files and the main functions that I need:

• searching for lines within the log file (and the return line of entries).
• pagination from string x to string y .

Now I checked Logstash, and it looked great for my first function (search), but not for the second. I was under the idea that I could somehow index the line number of the file along with the log information of each record, but I cannot find a way.

Is there any Logstash filter for this? or filebeat processor? I can not make it work.

I thought that perhaps I could create a way for all my processes to enter the database with the processed information, but this is also impossible (or very difficult) because the log handler also does not know what the current log line is.

In the end, what can I do to serve as a way to paginate my log file (through the service) would actually open it, go to a specific line and show it in the service, which is not very optimal, since the file can be very large and I'm already indexing it in Elasticsearch (using Logstash).

My current configuration is very simple:

Filebeat

 filebeat.prospectors: - type: log paths: - /path/of/logs/*.log output.logstash: hosts: ["localhost:5044"] 

Logstash

 input { beats { port => "5044" } } output { elasticsearch { hosts => [ "localhost:9200" ] } } 

Right now, for example, I get an element like:

  { "beat": { "hostname": "my.local", "name": "my.local", "version": "6.2.2" }, "@timestamp": "2018-02-26T04:25:16.832Z", "host": "my.local", "tags": [ "beats_input_codec_plain_applied", ], "prospector": { "type": "log" }, "@version": "1", "message": "2018-02-25 22:37:55 [mylibrary] INFO: this is an example log line", "source": "/path/of/logs/example.log", "offset": 1124 } 

If I could somehow include a line_number: 1 type field in this element, it would be great, since I could use Elasticsearch filters to actually navigate through all the logs.

---

If you have ideas on different ways to store my logs (and navigation), please let me know

Answer 1

Are log files created? Or can you change the structure of the magazine? You can then add the counter as a prefix and filter it with logstash.

For example, for

 12345 2018-02-25 22:37:55 [mylibrary] INFO: this is an example log line 

your filter should look like this:

 filter { grok { match => {"message" => "%{INT:count} %{GREEDYDATA:message}" overwrite => ["message"] } } 

A new "count" field will be created. Then you can use it for your own purposes.

Answer 2

At the moment, I don’t think there are any solutions here. Logstash, Beats, Kibana have ideas for events over time and basically how things are arranged. Line numbers are rather the functionality of a text editor.

To some extent, Kibana can show you the events in a file. This will not give you a list of pages by pages where you can actually click the page number, but using time frames that you could theoretically look at the whole file.

There are similar requests (improvements) for Beats and Logstash .

Answer 3

Let me first give you what is probably the main reason why Filebeat does not yet have a line number field. When Filebeat resumes reading the file (for example, after a restart), it returns fseek from the last recorded offset. If he had to report line numbers, he needed to either save this state in his registry, or re-read the file and recalculate the new line characters to the offset.

If you want to offer a service that allows you to break pages into magazines that are supported by Elasticsearch, you can use the scroll API with a request for the file. You should sort the results of @timestamp and then offset . Your service will use the scroll request to get the first page of results.

 POST /filebeat-*/_search?scroll=1m { "size": 10, "query": { "match": { "source": "/var/log/messages" } }, "sort": [ { "@timestamp": { "order": "asc" } }, { "offset": "asc" } ] } 

Then, to get all future pages, you use the scroll_id returned from the first request.

 POST /_search/scroll { "scroll" : "1m", "scroll_id" : "DnF1ZXJ5VGhlbkZldGNoBwAAAAAAPXDOFk12OEYw=" } 

This will give you all the log data for a specific file name, even tracking it through rotations. If line numbers are critical, you can produce them synthetically by counting events starting from the first event with offset == 0 , but I avoid this because it is very error prone, especially if you have ever added any kind of filtering or multi-line grouping.