Can ElasticSearch aggregation results be returned to the index?

Question

So, I have an index with ping data from host A to host B, and my data looks like this:

{ "@version" => "1", "@timestamp" => "2014-07-17T21:17:34.030Z", "host" => "host_a", "to_host" => "host_b", "value" => "25.6", "from_host" => "host_a", "stat_type" => "ping" } 

The goal is to also store the 90th percentile data for the ping value ("value" in the above) in a moving window, for example. last hour, last day, etc.

I know I can do this with aggregation, but my question is this:

Does ElasticSearch support a way to automatically add aggregation output (or query for it) back to the index?

I know that I could just take the output, set up the fields, and then return the data using a helper application, but I was curious if this could be used only with ES.

To use an equivalent SQL example, I would look for something like this:

 create table agg select id, count(*) as counts from data group by id; 

Answer 1

Maybe a little not what you are looking for, but you can do it using Logstash, which is part of Elasticsearch http://www.elasticsearch.com/blog/welcome-jordan-logstash/ .

I won’t go into details here (there are a lot of tutorials in Logstash), but adding the results of your aggregation request to the log file and having logstash automatically accepts the results and loads them into the Elasticsearch index is pretty trivial. This tutorial describes everything you need to read log statistics from a log file and automatically index content:

http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash

you could, of course, do this in code, but most of the work will be taken care of here, once you have the logstash setting set up, all you have to do is add your results to the log file.