Increment times out; set always succeeds after retry

Question

I'm getting strange behavior in memcached, in particular, behavior that is strange in its consistency. Here is my test:

@Test
public void testMemc() {
    logger.info("Setting head.");
    memc.set(env.memcachedQueueKeys().head, 3600, 0);
    logger.info("Set head; incrementing.");
    memc.incr(env.memcachedQueueKeys().head, 1);
    logger.info("Incremented.");
}

And here is the output:

28 11:04:52.932 INFO; Setting head.

2014-01-28 11:04:52.933 WARN net.spy.memcached.MemcachedConnection: Could not redistribute to another node, retrying primary node for q:unittest:scannedemails:w.

28 11:04:52.933 INFO; Set head; incrementing.

2014-01-28 11:04:52.935 WARN net.spy.memcached.MemcachedConnection: Could not redistribute to another node, retrying primary node for q:unittest:scannedemails:w.

FAILED: testMemc net.spy.memcached.OperationTimeoutException: Mutate operation timed out,unable to modify counter [q:unittest:scannedemails:w]
at net.spy.memcached.MemcachedClient.mutate(MemcachedClient.java:1484)
at net.spy.memcached.MemcachedClient.incr(MemcachedClient.java:1529)
at me.unroll.emailroller.ActOnScanResultsTest.testMemc(ActOnScanResultsTest.java:295)

Most of my intuition for this kind of error fails me here. The following things are all strange:

1. Why does it always fail exactly once to set?
2. Why does it permanently fail to increment after seeming to succeed at set?

This is on a high-load server (yes, it's a little wrong to be running a test on a load-bearing server, but if it catches issues like this there's at least some advantage). What can cause this consistent failure? There is only one node.

Answer 1

Problem is I couldn't connect at all. This is a bug in spymemcached, since the set operation did not throw an exception even though it had no memcached server to perform set on.