data stored by pylibmc cannot be uncompressed in php memcached

Question

We have an application where we are using python to store lots of data in memcached.We are using pylibmc in python and on php side we are using php-memcached library.As a summary

• pylibmc v.1.2.3
• php-memcached v.2.0.1
• libmemcached v1.0.8.

Everything else is fine except when compression comes into play. This is how data is compressed in python

import pylibmc

mem = pylibmc.Client(['10.90.15.104:11211'], binary=True)
mem.set('foo','this is a rather long string. this is a rather '+
'long string. this is a rather long string. this is a rather' + 
'long string. this is a rather long string', 0, 10)

checking in telnet we see some garbled value, which means it was compressed. Now reading it in php.

$memd = new Memcached();
$memd->addServer('10.90.15.104', 11211);
echo $memd->get('foo');

When above is run, we get same garbled value, which means it is not getting uncompressed. pylibmc is using zlib, so accordingly I have changed php's compression type to zlib also. What other setting needs to be done? Please help.

For further reference here are the memcached's output after setting the string in python pylibmc

get foo
VALUE foo 8 40
x+��,V�D��Ē��"����t�⒢̼t=���g\5#
END

And here's memcached's output for string stored using PHP's memcached client:

get foo
VALUE foo 48 44
�x�+��,V�D��Ē��"����t�⒢̼t=���g\5#
END

As you can see there's something fishy in this. Compressed size in pylibmc is 40 bytes and the same data compressed using php-memcached is of 44 bytes. Also notice the flags as 8 when stored using pylibmc and 48 when stored using php-memcached !

Answer 1

i think what you observe is due to the fact that memcache itself does not implement compression so each library does it in their own way, just compare the flags used to indicate compression is in use

as defined by pylibmc

#define PYLIBMC_FLAG_ZLIB (1 << 3) (so this is flag == 8)

and by php-memcached

#define MEMC_VAL_COMPRESSED (1<<4)

#define MEMC_VAL_COMPRESSION_ZLIB (1<<5)

#define MEMC_VAL_COMPRESSION_FASTLZ (1<<6)

so i think unless you are willing to modify one of this libraries to make it's flags consistent with the other then there is no way out

Edit: Ok, so here is the little patch that bring compression support for pylibmc and php-memcached in sync. See my github fork of pylibmc.

Big fat warring - it works only on strings, so if you want to store objects you MUST do de/serialization on your own (JSON).

Answer 2

Raber,

As you said I did changes

1. Changing php compression method from FASTLZ to ZLIB

2. Changing pyLibmc flag to 1 << 5 in _pylibmcmodule.h file and re-installed pylibmc

   #define PYLIBMC_FLAG_ZLIB (1 << 5)
   

Are these correct changes? Something more needs to be done? Because Its not working, getting following error

Warning: Memcached::get(): unknown payload type in uaTestMemcached.php on line 5

bool(false)