Elastic search: RemoteTransportException paginated to search for more than 10,000 results

Question

I use Elastic Search to run paginated index searches through a query given in my java program. Here I have two cases, as follows:

Search output using ES scroll.

Here For example: the total search result is 10,010, the page size is 100. Therefore, the search result will be divided into 11 pages with 100 entries each. When I look at my result on each page up to page 10, the records are returned correctly, i.e. for the first 10,000 entries. But when I look at the 11th page of ie entries from 10 001 to 10 010, I get below error:

RemoteTransportException [[James Jaspers] [127.0.0.1:9300] [indexes: data / read / search [phase / query + fetch]]]; inested: QueryPhaseExecutionException [The results window is too large, from + the size should be less than or equal to: [10000], but it was [10010].
Reason: QueryPhaseExecutionException [The results window is too large, from + the size must be less than or equal to: [10000], but it was [10010]. See Scrolling api for a more efficient way to request large datasets. This limit can be set by changing the index level parameter [index.max_result_window]

Below is a snippet of code, in this page of the search page the value 100 is passed, and DEFAULT_SEARCH_PAGE_SIZE is 1000

if (searchPage != null) {
            builder.setFrom((int) searchPage.getPageStart());
            builder.setSize((int) searchPage.getPageSize());
        } else {
            builder.setFrom(0);
            builder.setSize(DEFAULT_SEARCH_PAGE_SIZE);
        }

builder.setTypes(getType());
SearchResponse response = builder.execute().actionGet(60000);
SearchHits hits = response.getHits();
if (hits.getTotalHits() > 0) {
        for (SearchHit hit : response.getHits()) {
                    //process my hits and add them to list

        }
  }
//return the list

, , . , , , . 100 .

    if (searchPage != null) {
                builder.setFrom((int) searchPage.getPageStart());
                builder.setSize((int) searchPage.getPageSize());
            } else {
                builder.setFrom(0);
                builder.setSize(DEFAULT_SEARCH_PAGE_SIZE);
            }
           builder.setTypes(getType()).setScroll(new TimeValue(60000));

            SearchResponse response = builder.execute().actionGet(60000);
            SearchHits hits = response.getHits();
            if (hits.getTotalHits() > 0) {
                    for (SearchHit hit : response.getHits()) {
                        //process my hits and add them to list
                        }
                    }  
           //return the result

, Elastic Search Scroll API scrollId , , Ill , .. 1, 2, 3, 4... - , : 1 5, Scroll Api ?

adityasinghraghav

( 10 ), .. 10000-10010 elasticsearch 10010, , 10000

max_result_window. 10 000, . , , . , , .

, = 500 000, Max Result Windows Size = 100 000 = 1000.

5- , :

• 100 000 _i.e , 100 000 4 000, 1000.

• , 5 000 , 5 000, 4 000, 1 000?

Answer 1

, elasticsearch 10000. , ( 10 ), . 10000-10010 elasticsearch 10010, , 10000, 10 , , . , , 10000 . :

curl -XPUT http://1.2.3.4:9200/index/_settings -d '{ "index" : { "max_result_window" : 1000000}}'

api, , from , size -. api "size", , size 10, 5 , elasticsearch 50 . api , , "" . , . api .

- , : 1 5

, api , .

, elasticsearch , , - , , .

Answer 2

max_result_window. - , , .

, , 20 ( , 1 5 ), - 20- . , 20 , Out Of Memory , , , .

Search After (https://www.elastic.co/guide/en/elasticsearch/reference/5.1/search-request-search-after.html), . Search after , .