Items set with spymemcached cannot be fetched with php memcached

Question

I am using spymemcached. I set a couple of items. Then I run a php script, however then I cannot get all those items using php memcached. PHP-Memcached can only partially retrieve those items.

I cannot change php's hashing algorithm or distribution strategy. In our system we are using default hashing (which is jenkin's one-at-a-time according to php.net documentation). And distribution strategy is modulo for php-memcached. I have read that spymemcached uses consistent hashing. Is there any way by which I can use modulo hashing in spymemcached.

In other words how can I make spymemcached's set operations or any other store operations compatible with php-memcached's get operations?

If spymemcached is not able to do that, are there any other memcached client in java that will allow me to do so?

Help will not only be appreciated, it will also be rewarded a bounty.

Java code:

public static void main(String [] args) {
    List<InetSocketAddress> addrs = new ArrayList<>();
    addrs.add(new InetSocketAddress("10.90.12.87", 11211));
    addrs.add(new InetSocketAddress("10.90.12.87", 11311));
    try {
        MemcachedClient memcache = new MemcachedClient(addrs);
        memcache.add("foo", 0, "bar");
        memcache.add("sample", 0, "key");
        memcache.add("try", 0, "another");
        memcache.add("ax-spadg-list", 0, "3045,6645");
    } catch (IOException ex) {
        Logger.getLogger(CategoryDataOperator.class.getName()).log(Level.SEVERE, null, ex);
    }
    System.out.println("Done");
}

PHP code:

<?php
$mem = new Memcached();
$mem->addServer('10.90.12.87', 11211);
$mem->addServer('10.90.12.87', 11311);
var_dump $mem->get('foo');
var_dump($mem->get('try'));
var_dump($mem->get('sample'));
var_dump($mem->get('ax-spadg-list'));

Answer 1

The problem is about Hash, the default php-memcached hash is

(Jenkins one-at-a-time) item key hashing algorithm

whereas the list of spymemcached hashes are:

• NATIVE_HASH: simply Native hash (String.hashCode()). does not match with default php-memcached Memcached::HASH_DEFAULT
• CRC_HASH => Memcached::HASH_CRC
• FNV1_64_HASH => Memcached::HASH_FNV1_64
• FNV1A_64_HASH => Memcached::HASH_FNV1A_64
• FNV1_32_HASH => Memcached::HASH_FNV1_32
• FNV1A_32_HASH => Memcached::HASH_FNV1A_32
• KETAMA_HASH => "MD5-based hash algorithm used by ketama." So maybe Memcached::HASH_MD5 but anyway is not Memcached::HASH_DEFAULT

So there is not direct matching between the two libs if you can't change PHP client configuration or extend spymemcached lib.

SOLUTION 1: If you look on history (you can have an example with php client hash modification).

SOLUTION 2: Else you can create a JenkinHash class (I copy past the Xmemcached code: https://code.google.com/p/xmemcached/source/browse/trunk/src/main/java/net/rubyeye/xmemcached/HashAlgorithm.java?r=801#176 [But take in consideration the Xmemcached Licenses and keep the author/license inside the source code])

import net.spy.memcached.HashAlgorithm;

import java.io.UnsupportedEncodingException;

public class JenkinsHash implements HashAlgorithm {
    @Override
    public long hash(String k) {
        try {
            int hash = 0;
            for (byte bt : k.getBytes("utf-8")) {
                hash += (bt & 0xFF);
                hash += (hash << 10);
                hash ^= (hash >>> 6);
            }
            hash += (hash << 3);
            hash ^= (hash >>> 11);
            hash += (hash << 15);
            return hash;
        } catch (UnsupportedEncodingException e) {
            throw new IllegalStateException("Hash function error", e);
        }
    }
}

then:

import net.spy.memcached.*;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.ArrayList;
import java.util.List;
import java.util.logging.Level;
import java.util.logging.Logger;

public class Main {

    public static void main(String[] args) throws IOException {
        List<InetSocketAddress> addrs = new ArrayList<InetSocketAddress>();
        addrs.add(new InetSocketAddress("127.0.0.1", 11211));
        addrs.add(new InetSocketAddress("172.28.29.22", 11211));
        try {
            ConnectionFactory connectionFactory = new ConnectionFactoryBuilder()
                .setProtocol(ConnectionFactoryBuilder.Protocol.TEXT)
                .setHashAlg(new JenkinsHash())
                .setLocatorType(ConnectionFactoryBuilder.Locator.ARRAY_MOD).build();
            MemcachedClient memcache = new MemcachedClient(connectionFactory, addrs);
            memcache.add("foo", 0, "bar2");
            memcache.add("sample", 0, "key");
            memcache.add("try", 0, "another");
            memcache.add("ax-spadg-list", 0, "3045,6645");
        } catch (IOException ex) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
        }
        System.out.println("Done");
    }
}

With php script:

<?php

$memcached = new Memcached();
$memcached->addserver('127.0.0.1', 11211);
$memcached->addserver('172.28.29.22', 11211);
var_dump($memcached->get('foo'));
var_dump($memcached->get('try'));
var_dump($memcached->get('sample'));
var_dump($memcached->get('ax-spadg-list'));

test:

$ echo "flush_all" | nc 172.28.29.22 11211 && echo "flush_all" | nc 127.0.0.1 11211
OK
OK
$ php mem.php 
bool(false)
bool(false)
bool(false)
bool(false)

RUN JAVA

$ php mem.php 
string(4) "bar2"
string(7) "another"
string(3) "key"
string(9) "3045,6645"

SOLUTION 3: use https://code.google.com/p/xmemcached/ with ONE_AT_A_TIME hash algorithm

import net.rubyeye.xmemcached.HashAlgorithm;
import net.rubyeye.xmemcached.MemcachedClient;
import net.rubyeye.xmemcached.MemcachedClientBuilder;
import net.rubyeye.xmemcached.XMemcachedClientBuilder;
import net.rubyeye.xmemcached.exception.MemcachedException;
import net.rubyeye.xmemcached.impl.ArrayMemcachedSessionLocator;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeoutException;

public class Main {

    public static void main(String[] args) throws IOException, InterruptedException, MemcachedException, TimeoutException {
        List<InetSocketAddress> addrs = new ArrayList<InetSocketAddress>();
        addrs.add(new InetSocketAddress("127.0.0.1", 11211));
        addrs.add(new InetSocketAddress("172.28.29.22", 11211));
        MemcachedClientBuilder builder = new XMemcachedClientBuilder(addrs);
        builder.setSessionLocator(new ArrayMemcachedSessionLocator(HashAlgorithm.ONE_AT_A_TIME));
        MemcachedClient memcachedClient = builder.build();
        memcachedClient.set("foo", 0, "bar2");
        memcachedClient.set("sample", 0, "key");
        memcachedClient.set("try", 0, "another");
        memcachedClient.set("ax-spadg-list", 0, "3045,6645");
        memcachedClient.shutdown();
        System.out.println("Done");
    }
}

Answer 2

Hash algorithms that spymemcached supports are here: https://github.com/couchbase/spymemcached/blob/master/src/main/java/net/spy/memcached/DefaultHashAlgorithm.java

You should be able to change the hash algorithm by using a ConnectionFactory to create your MemcachedClient. Do something like this:

ConnectionFactoryBuilder builder = new ConnectionFactoryBuilder();
builder.setHashAlgorithm(HashAlgorithm.CRC_HASH);
ConnectionFactory factory = builder.build();
MemcachedClient client = new MemcachedClient(Arrays.asList(new InetSocketAddr("localhost", 11211)), factory);

Answer 3

Re: Kakawait (and also to Shades88)

Solution #2 is incorrect because xmemcached didn't properly port the original C code of the Jenkins hash, which uses an unsigned. Fixing this will also solve the ArrayIndexOutOfBoundsException that Shades88 was seeing.

public class JenkinsHash implements HashAlgorithm {
    @Override
    public long hash(String k) {
        try {
            int hash = 0;
            for (byte bt : k.getBytes("utf-8")) {
                hash += (bt & 0xFF);
                hash += (hash << 10);
                hash ^= (hash >>> 6);
            }
            hash += (hash << 3);
            hash ^= (hash >>> 11);
            hash += (hash << 15);

            // the hash variable in the original C code is a uint32.
            // convert the java signed int to an "unsigned",
            // represented via a long:
            return hash & 0xFFFFFFFFl;
        } catch (UnsupportedEncodingException e) {
            throw new IllegalStateException("Hash function error", e);
        }
    }
}

// Unit test
public class JenkinsHashTest {
    @Test
    public void testHash() throws Exception {
        JenkinsHash j = new JenkinsHash();
        Properties p = new Properties();

        // This file contains k/v mappings,
        // with values generated by the reference C code
        p.load(new FileReader("src/test/resources/jenkinsHashTest.dat"));

        for (Entry<Object, Object> entry : p.entrySet()) {
            long result = j.hash((String)entry.getKey());
            // Print out hash mismatches
            if (result != Long.parseLong((String)entry.getValue())) {
                System.out.println("Key: " + (String)entry.getKey());
                System.out.println("Expected Hash Value: " + Long.parseLong((String)entry.getValue()));
                System.out.println("Actual Hash Value: " + result);
            }
            assertEquals(result, Long.parseLong((String)entry.getValue()));
        }
    }
}

The test data file is to compare the Java code against the C code. Build the C code then hash a bunch of random words and map them in a file, like this:

jenkinsHashTest.dat:

sausage=2834523395
blubber=1103975961
pencil=3318404908
cloud=670342857
moon=2385442906
water=3403519606
computer=2375101981
school=1513618861
network=2981967937
hammer=1218821080

... add as many as you want