Increase in waiting time; set will always succeed after retrying

Question

I get strange behavior in memcached, in particular, behavior that is strange in its sequence. Here is my test:

@Test public void testMemc() { logger.info("Setting head."); memc.set(env.memcachedQueueKeys().head, 3600, 0); logger.info("Set head; incrementing."); memc.incr(env.memcachedQueueKeys().head, 1); logger.info("Incremented."); } 

And here is the conclusion:

28 11: 04: 52,932 INFO; Head installation.

2014-01-28 11: 04: 52.933 WARN net.spy.memcached.MemcachedConnection: Could not reallocate another node by repeating the primary node for q: unittest: scannedemails: w.

28 11: 04: 52,933 INFO; Install the head; increment.

2014-01-28 11: 04: 52.935 WARN net.spy.memcached.MemcachedConnection: Unable to reallocate another node by repeating the primary node for q: unittest: scannedemails: w.

FAILED: testMemc net.spy.memcached.OperationTimeoutException: the mutate operation is disabled, the counter could not be changed [q: unittest: scannedemails: w]
in net.spy.memcached.MemcachedClient.mutate (MemcachedClient.java:1484)
in net.spy.memcached.MemcachedClient.incr (MemcachedClient.java:1529)
at me.unroll.emailroller.ActOnScanResultsTest.testMemc (ActOnScanResultsTest.java:295)

Most of my intuition for this kind of error does not allow me here. Everything is strange:

• Why does it always work exactly once for installation?
• Why can't it constantly increase after it seems successful at tuning?

This is on a server with a high load (yes, it’s a little wrong to run the test on the host server, but if it catches problems like this, at least some advantage). What could cause this sequential failure? There is only one node.

Answer 1

The problem is that I could not connect at all. This is an error in spymemcached because the set operation did not throw an exception, even if it did not have a memcached server to do set on.