Enter aggregation results in the index

Question

The goal is to create an Elasticsearch index with only the most recent documents in groups of related documents , in order to keep track of the current state of some control counters and states.

I created a simple Elasticsearch aggregation request :

{ "size": 0, "aggs": { "group_by_monitor": { "terms": { "field": "monitor_name" }, "aggs": { "get_latest": { "top_hits": { "size": 1, "sort": [ { "timestamp": { "order": "desc" } } ] } } } } } } 

He groups related documents into buckets and selects the most recent document for each bucket.

Here are the different ideas that I had to fulfill:

• use the aggregate query directly to push the results to the index, but this is not possible: Is it possible to return the results of the ElasticSearch aggregation to the index?
• use the Logstash Elasticsearch input plugin to execute the aggregation request and the elasticsearch output Plugin to insert into the index, but it looks like the input plugin looks only at the hits field and cannot process the aggregation results: Aggressive request Possible input of the ES plugin !
• use the Logstash http_poller plugin to get the JSON document, but it doesn't seem to allow you to specify the body for the HTTP request!
• use the Logstash exec plugin to execute cURL commands to get JSON, but that seems pretty cumbersome, and my last resort.
• use the NEST API to create a basic application that will poll, retrieve the results, clear them and insert the resulting documents into the target index, but I would like to avoid adding a new support tool.

Is there a fairly sophisticated way to accomplish this?

Answer 1

Edit the logstash.conf file as follows

 input { elasticsearch { hosts => "localhost" index => "source_index_name" type =>"index_type" query => '{Query}' size => 500 scroll => "5m" docinfo => true } } output { elasticsearch { index => "target_index_name" document_id => "%{[@metadata][_id]}" } }