First of all, the native library must be preinstalled on all worker nodes. Path to that library must be specified in spark-env.sh
:
export SPARK_LIBRARY_PATH=/path/to/native/library
SPARK_PRINT_LAUNCH_COMMAND
environment variable might be used to diagnose it:
export SPARK_PRINT_LAUNCH_COMMAND=1
If everything's set correctly, you will see output like this:
Spark Command:
/path/to/java -cp <long list of jars> -Djava.library.path=/path/to/native/library <etc>
========================================