How to use scroll in Elasticsearch aggregation?

Question

I am using Elasticsearch 5.3. I collect data from some data, but there are too many results to return in a single query. I tried using size = Integer.MAX_VALUE; but even that turned out to be less. There is a scroll method in the ES search API through search results. Is there a similar function for the org.elasticsearch.search.aggregations.AggregationBuilders.terms aggregator and how to use it? Can I use the search scroll API for aggregators?

Answer 1

In ES 5.3, you can partition the terms buckets and retrieve one partition for each request.

For example, in the query below, you can request a breakdown of your buckets into 10 sections and only return the first section. It will return ~ 10x less data than if you wanted to get all the buckets at once.

 { "size": 0, "aggs": { "my_terms": { "terms": { "field": "my_field", "include": { "partition": 0, "num_partitions": 10 }, "size": 10000 } } } } 

Then you can make a second request by increasing the section to 1, etc.

 { "size": 0, "aggs": { "my_terms": { "terms": { "field": "my_field", "include": { "partition": 1, <--- increase this up until partition 9 "num_partitions": 10 }, "size": 10000 } } } } 

To add this to your Java code, you can do it like this:

 TermsAggregationBuilder agg = AggregationBuilders.terms("my_terms"); agg.includeExclude(new IncludeExclude(0, 10));