You need to have the .so file on your class-path as well.
You can download the pre-built binaries from http://rxtx.qbang.org/wiki/index.php/Download unfortunately they are only available in 32-bit.
If your using Eclipse you can just drop the .so files in the root of your project and it should work. If that doesn't work you could try setting the java.library.path VM argument. Instructions for that are available at How to set the java.library.path from Eclipse