Lucene: Why does the search cost increase with page size?

Question

I am using ElasticSearch cluster with 1G cache for each machine. I configured the search engine to 5000 / perpage.

When I search for page 3, it costs about 400 ms. But when I searched for page 300, the cost increased to over 5000 ms! About 60% of the cache was free in this situation.

Why has the cost increased more than 10 times?

Answer 1

To publish this page, Elasticsearch must extract all previous pages, just to drop them at the end. Searching with is from: 1000, size: 100equivalent to searching using from: 0, size: 100100, only you get only the last 100 results. It also means that every document must be jammed, which is a potentially expensive operation.

There has been a recent optimization that should improve performance in cases where you use search scroll , see this is github problem .