Scan and scroll using the JEST API

Question

I am currently working with JEST: https://github.com/searchbox-io/Jest

Can I scan and scroll using this API?

http://www.elasticsearch.org/guide/reference/api/search/search-type/

I am currently using the Search command:

Search search = new Search("{\"size\" : "+RESULT_SIZE+", \"query\":{\"match_all\":{}}}"); 

but I'm worried about large result sets. If you use the "Search" command for this, how do you set the arguments "search_type = scan & scroll = 10m & size = 50"?

Answer 1

EDIT:

It doesn't look like JEST currently supports the scan type search: In an evil quick turn, it seems like JEST now supports scan type search! Supports @Ferhat for a quick turn! JEST - SearchType.java

---

Have you just considered using the ElasticSearch Transport client? I could understand if you like the JEST API a little better, but as new functions for ElasticSearch appear ( Illustration A: ElasticSearch 0.90 is fantastic! ), You will receive them as soon as they pop up, rather than waiting for JEST to catch up .

My $ 0.02.

Answer 2

Can I scan and scroll using this API?

Yes it is. My implementation works like this.

Start your search for elasticity scrolling:

  public SearchResult startScrollSearch (String type, Long size) throws IOException { String query = ConfigurationFactory.loadElasticScript("my_es_search_script.json"); Search search = new Search.Builder(query) // multiple index or types can be added. .addIndex("myIndex") .addType(type) .setParameter(Parameters.SIZE, size) .setParameter(Parameters.SCROLL, "1m") .build(); SearchResult searchResult = EsClientConn.getJestClient().execute(search); return searchResult; } 

The SearchResult object will return the first (size), which, as usual, will cancel the search, but will return to the scrollId parameter, which is a link to stay in the result set, which will be stored in memory for you. Parameters.SCROLL, will determine the time during which this search will be stored in memory.

To read scrollId:

 scrollId = searchResult.getJsonObject().get("_scroll_id").getAsString(); 

To get more detailed information from the result set, you should use something like the following:

 public JestResult readMoreFromSearch(String scrollId, Long size) throws IOException { SearchScroll scroll = new SearchScroll.Builder(scrollId, "1m") .setParameter(Parameters.SIZE, size).build(); JestResult searchResult = EsClientConn.getJestClient().execute(scroll); return searchResult; } 

Do not forget that every time you read the result, a new scrollId from elastic is returned.

Please tell me if you have any doubts.

Answer 3

I agree that we need to catch up, however, please open the problem if you need a function.

Please check https://github.com/searchbox-io/Jest/blob/master/jest/src/test/java/io/searchbox/core/SearchScrollIntegrationTest.java in the main