ElasticSearch (search_context_missing_exception) with Search :: ElasticSearch :: Scroll

Question

I use Search :: Elasticsearch and Search :: Elasticsearch :: Scroll to search and scroll on my elasticsearch server.

While scrolling for some queries, I see the following errors when scrolling through search results:

2016/03/22 11:03:38 - 265885 FATAL: [Daemon.pm][8221]: Something gone wrong, error $VAR1 = bless( { 'msg' => '[Missing] ** [http://localhost:9200]-[404] Not Found, called from sub Search::Elasticsearch::Scroll::next at searcher.pl line 92. With vars: {\'body\' => {\'hits\' => {\'hits\' => [],\'max_score\' => \'0\',\'total\' => 5215},\'timed_out\' => bless( do{\\(my $o = 0)}, \'JSON::XS::Boolean\' ),\'_shards\' => {\'failures\' => [{\'index\' => undef,\'reason\' => {\'reason\' => \'No search context found for id [4920053]\',\'type\' => \'search_context_missing_exception\'},\'shard\' => -1},{\'index\' => undef,\'reason\' => {\'reason\' => \'No search context found for id [5051485]\',\'type\' => \'search_context_missing_exception\'},\'shard\' => -1},{\'index\' => undef,\'reason\' => {\'reason\' => \'No search context found for id [4920059]\',\'type\' => \'search_context_missing_exception\'},\'shard\' => -1},{\'index\' => undef,\'reason\' => {\'reason\' => \'No search context found for id [5051496]\',\'type\' => \'search_context_missing_exception\'},\'shard\' => -1},{\'index\' => undef,\'reason\' => {\'reason\' => \'No search context found for id [5051500]\',\'type\' => \'search_context_missing_exception\'},\'shard\' => -1}],\'failed\' => 5,\'successful\' => 0,\'total\' => 5},\'_scroll_id\' => \'c2NhbjswOzE7dG90YWxfaGl0czo1MjE1Ow==\',\'took\' => 2},\'request\' => {\'serialize\' => \'std\',\'path\' => \'/_search/scroll\',\'ignore\' => [],\'mime_type\' => \'application/json\',\'body\' => \'c2Nhbjs1OzQ5MjAwNTM6bHExbENzRDVReEc0OV9UMUgzd3Vkdzs1MDUxNDg1OnJrQ3lsUkRKVHRxRWRWeURoOTB4WVE7NDkyMDA1OTpscTFsQ3NENVF4RzQ5X1QxSDN3dWR3OzUwNTE0OTY6cmtDeWxSREpUdHFFZFZ5RGg5MHhZUTs1MDUxNTAwOnJrQ3lsUkRKVHRxRWRWeURoOTB4WVE7MTt0b3RhbF9oaXRzOjUyMTU7\',\'qs\' => {\'scroll\' => \'1m\'},\'method\' => \'GET\'},\'status_code\' => 404} ', 'stack' => [ [ 'searcher.pl', 92, 'Search::Elasticsearch::Scroll::next' ] ], 'text' => '[http://localhost:9200]-[404] Not Found', 'vars' => { 'body' => { 'hits' => { 'hits' => [], 'max_score' => '0', 'total' => 5215 }, 'timed_out' => bless( do{\(my $o = 0)}, 'JSON::XS::Boolean' ), '_shards' => { 'failures' => [ { 'index' => undef, 'reason' => { 'reason' => 'No search context found for id [4920053]', 'type' => 'search_context_missing_exception' }, 'shard' => -1 }, { 'index' => undef, 'reason' => { 'reason' => 'No search context found for id [5051485]', 'type' => 'search_context_missing_exception' }, 'shard' => -1 }, { 'index' => undef, 'reason' => { 'reason' => 'No search context found for id [4920059]', 'type' => 'search_context_missing_exception' }, 'shard' => -1 }, { 'index' => undef, 'reason' => { 'reason' => 'No search context found for id [5051496]', 'type' => 'search_context_missing_exception' }, 'shard' => -1 }, { 'index' => undef, 'reason' => { 'reason' => 'No search context found for id [5051500]', 'type' => 'search_context_missing_exception' }, 'shard' => -1 } ], 'failed' => 5, 'successful' => 0, 'total' => 5 }, '_scroll_id' => 'c2NhbjswOzE7dG90YWxfaGl0czo1MjE1Ow==', 'took' => 2 }, 'request' => { 'serialize' => 'std', 'path' => '/_search/scroll', 'ignore' => [], 'mime_type' => 'application/json', 'body' => 'c2Nhbjs1OzQ5MjAwNTM6bHExbENzRDVReEc0OV9UMUgzd3Vkdzs1MDUxNDg1OnJrQ3lsUkRKVHRxRWRWeURoOTB4WVE7NDkyMDA1OTpscTFsQ3NENVF4RzQ5X1QxSDN3dWR3OzUwNTE0OTY6cmtDeWxSREpUdHFFZFZ5RGg5MHhZUTs1MDUxNTAwOnJrQ3lsUkRKVHRxRWRWeURoOTB4WVE7MTt0b3RhbF9oaXRzOjUyMTU7', 'qs' => { 'scroll' => '1m' }, 'method' => 'GET' }, 'status_code' => 404 }, 'type' => 'Missing' }, 'Search::Elasticsearch::Error::Missing' ); 

The code I'm using is the following (simplified):

 # Retrieve scroll my $scroll = $self->getScrollBySignature($item); # Retrieve all affected documents ids while (my @docs = $scroll->next(500)) { # Do stuff with @docs } 

The getScrollBySignature function has the following code to call the elasticSearch function

 my $scroll = $self->{ELASTIC}->scroll_helper( index => $self->{INDEXES}, search_type => 'scan', ignore_unavailable => 1, body => { size => $self->{PAGINATION}, query => { filtered => { filter => { bool => { must => [{term => {signature_id => $item->{profileId}}}, {terms => {channel_type_id => $type}}] } } } } } ); 

As you can see, I am scrolling without missing a scroll option, then, as the documentation says, the time that the scroll is alive is 1 minute.

Elastic search is a cluster of 3 servers, and a query that ends with this error retrieves a bit of more than 5000 documents.

My first solution was to update the lifetime to scroll to 5 minutes, and the error did not appear.

The question is that, as I understand it, every time I call $ scroll-> next (), the lifetime from the scroll is affected updated by 1 m more, then how can I get these context-related errors?

Am I doing something bad?

Thanks to everyone.

Answer 1

Well, a good rule of thumb is inside the block ->next() , don't stop iterating more than the time you set up in the scroll.

Between each call ->next() you cannot stay longer than this configured. If you stay longer, the scroll may not be there, and an earch_context_missing_exception error will appear.

My solution for this problem was inside the next block only for storing data in an array / hash structure and after completing the scroll process with all the data.

Solution of the problem:

  # Retrieve scroll my $scroll = $self->getScrollBySignature($item); # Retrieve all affected documents ids my @allDocs; while (my @docs = $scroll->next(500)) { push @allDocs, map {$_->{_id}} @docs } foreach (@allDocs) { # Do stuff with doc } 

Answer 2

The first thing that comes to mind is that the timer is not updated. Have you checked this? You can make a request every 10 seconds, for example, and see if there is an error in the 6th request ...