libmemcached - memcached_mget seems to block

Question

I have a single memcached server. I use the libmemcached C api to interface with it. I am using non blocking mode and no reply mode (behavior flags).

I am also using libmemcached inside an nginx C module if that helps.

The problem is that memcached_mget seems to block, ie for a server which has a latency of about 40ms, memcached_mget takes 40ms to complete. This is not exactly async.

Here is the code that I am using:

const char* localKeys[2] = {"key1", "key2"};
size_t k_length[2] = {4, 4};

gettimeofday(&t1, NULL);
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, log, 0, "no block: %d ", memcached_behavior_get(memc, MEMCACHED_BEHAVIOR_NO_BLOCK));
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, log, 0, "no reply: %d ", memcached_behavior_get(memc, MEMCACHED_BEHAVIOR_NOREPLY));
retFt = memcached_mget(memc, localKeys, k_length, 2);
gettimeofday(&t2, NULL);

ngx_log_debug2(NGX_LOG_DEBUG_HTTP, log, 0, "memcached_mget duration (microseconds): %d, result : %d", t2.tv_usec - t1.tv_usec, retFt);

retFt = memcached_fetch_execute(memc, fp, &callback_struct, 1);

I am using ASCII protocol. Tried using binary also, but it didn't help. I also tried memcached_mget_execute. I am simulating the latency using tc, like so :

tc qdisc add dev lo root netem delay 20ms

Answer 1

libmemcached's API is always blocking. The "nonblocking" mode just sets that flag on the socket. If you use that flag, libmemcached does its own blocking by calling poll().