Couchbase: net.spy.memcached.internal.CheckedOperationTimeoutException

Question

I am loading a local Couchbase instance with specific application json objects.

Relevant Code:

CouchbaseClient getCouchbaseClient() { List<URI> uris = new LinkedList<URI>(); uris.add(URI.create("http://localhost:8091/pools")); CouchbaseConnectionFactoryBuilder cfb = new CouchbaseConnectionFactoryBuilder(); cfb.setFailureMode(FailureMode.Retry); cfb.setMaxReconnectDelay(1500); // to enqueue an operation cfb.setOpTimeout(10000); // wait up to 10 seconds for an operation to succeed cfb.setOpQueueMaxBlockTime(5000); // wait up to 5 seconds when trying to // enqueue an operation return new CouchbaseClient(cfb.buildCouchbaseConnection(uris, "my-app-bucket", "")); } 

A way to save a record (I use sentences from Mass loading and exponential cancellation ):

  void continuosSet(CouchbaseClient cache, String key, int exp, Object value, int tries) { OperationFuture<Boolean> result = null; OperationStatus status = null; int backoffexp = 0; do { if (backoffexp > tries) { throw new RuntimeException(MessageFormat.format("Could not perform a set after {0} tries.", tries)); } result = cache.set(key, exp, value); try { if (result.get()) { break; } else { status = result.getStatus(); LOG.warn(MessageFormat.format("Set failed with status \"{0}\" ... retrying.", status.getMessage())); if (backoffexp > 0) { double backoffMillis = Math.pow(2, backoffexp); backoffMillis = Math.min(1000, backoffMillis); // 1 sec max Thread.sleep((int) backoffMillis); LOG.warn("Backing off, tries so far: " + tries); } backoffexp++; } } catch (ExecutionException e) { LOG.error("ExecutionException while doing set: " + e.getMessage()); } catch (InterruptedException e) { LOG.error("InterruptedException while doing set: " + e.getMessage()); } } while (status != null && status.getMessage() != null && status.getMessage().indexOf("Temporary failure") > -1); } 

When the continuosSet method calls a large number of objects for storage (one thread), for example.

 CouchbaseClient cache = getCouchbaseClient(); do { SerializableData data = queue.poll(); if (data != null) { final String key = data.getClass().getSimpleName() + data.getId(); continuosSet(cache, key, 0, gson.toJson(data, data.getClass()), 100); ... 

it throws a CheckedOperationTimeoutException inside the continuosSet method in the result.get () operation.

 Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: 127.0.0.1/127.0.0.1:11210 at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:160) ~[spymemcached-2.8.12.jar:2.8.12] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:133) ~[spymemcached-2.8.12.jar:2.8.12] 

Can someone shed light on this, how to overcome and recover from this situation? Is there a good technique / solution to the problem of loading into a Java client for Couchbase? I have already studied the Mass Execution Dialing documentation, which, unfortunately, is for the PHP Couchbase client.

Answer 1

My suspicion is that you can run this in a JVM created from a command line that does not have so much memory. If this happens, you can press longer pauses of the GC, which can cause the timeout that you mention.

I think it’s best to try a couple of things. First, add the -Xmx argument to the JVM to use more memory. See if the timeout happens later or leaves. If so, then my suspicion of memory is correct.

If this does not work, raise setOpTimeout () and see if it reduces the error or makes it go away.

Also, make sure that you are using the latest client.

By the way, I do not think that this is directly related to bulk upload. This may happen due to high resource consumption during bulk loading, but it seems like regular shutdown should work, or you never click on it.