Can I use Snappy compression from a local client without a full Hadoop install?

Question

I have a hadoop cluster that I access from my local machine for development/debugging. I don't have hadoop installed in my local box; instead I just use the client libraries (via Maven, FWIW).

I'm currently trying to read some SequenceFiles compressed with Snappy compression. I get an error: "native snappy library not available". So far, all of the information I can find about resolving this looks like it assumes you have a full Hadoop install on the client machine. They all talk about putting libhadoop.so and libsnappy.so in lib/native under your Hadoop directory. On my dev machine, I have no Hadoop directory.

While I suppose I could do a full install of Hadoop, I'd love to know if it's possible to do something simpler. So far, various attempts at getting my process to discover those libraries has failed. I may have just not hit upon the correct config yet, but on the basis of all the documentation and threads surrounding use of Snappy, I'm not sure it's possible (or at least straightforward).

Answer 1

I finally got this sorted out.

As shutty expected, it was just a matter of the class loader looking in the wrong spot. (Or if you want to think of it the other way, I didn't put it in the right spot.)

Good old strace ended up saving the day. It revealed the following:

stat64("/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/libsnappy.so", 0xb68eef50) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/server/libsnappy.so", 0xb68eef50) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/libsnappy.so", 0xb68eef50) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/i386/libsnappy.so", 0xb68eef50) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/intellij-12.1/bin/libsnappy.so", 0xb68eef50) = -1 ENOENT (No such file or directory)
stat64("./libsnappy.so", 0xb68eef50)    = -1 ENOENT (No such file or directory)
stat64("/usr/java/packages/lib/i386/libsnappy.so", 0xb68eef50) = -1 ENOENT (No such file or directory)
stat64("/lib/libsnappy.so", 0xb68eef50) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/libsnappy.so", 0xb68eef50) = -1 ENOENT (No such file or directory)

Once I just slapped the files into one of those spots, things were golden.

I hope this helps someone else out in the future.

Answer 2

Hadoop looks for libsnappy.so in $HADOOP_HOME/lib/native/. So you can try to do this:

1. Create a folder structure like /home/user/hadoop/lib/native
2. Put a libsnappy.so and libhadoop.so in this folder. You can copy it from your cluster.
3. Set $HADOOP_HOME to /home/user/hadoop
4. Run your app.