Spark netlib-java BLAS

Question

I am trying to troubleshoot a non-working installation of apache and netlib, and I don't know what to do next.

Here is some info:

• Spark 1.3.1 (but also tried 1.5.1)
• Three-node Mesos cluster

• Ubuntu Trusty on every node and installed BLAS package

  $ dpkg -l | grep 'blas\|atlas\|lapack' ii libopenblas-base 0.2.8-6ubuntu1 amd64 Optimized BLAS (linear algebra) library based on GotoBLAS2 $ update-alternatives --get-selections | grep 'blas\|lapack' libblas.so.3 auto /usr/lib/openblas-base/libblas.so.3 

I built a jar sample for testing if netlib-java can detect these libraries with the following code:

 object Main extends App { println(com.github.fommil.netlib.BLAS.getInstance().getClass().getName()) println(com.github.fommil.netlib.LAPACK.getInstance().getClass().getName()) } 

When I execute this code, I get the following response:

 $ java -jar artifacts/BLAStest-assembly-1.0.jar Mar 29, 2016 3:43:33 PM com.github.fommil.netlib.BLAS <clinit> WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS Mar 29, 2016 3:43:33 PM com.github.fommil.jni.JniLoader liberalLoad INFO: successfully loaded /tmp/jniloader6790966128222263615netlib-native_ref-linux-x86_64.so com.github.fommil.netlib.NativeRefBLAS Mar 29, 2016 3:43:33 PM com.github.fommil.netlib.LAPACK <clinit> WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK Mar 29, 2016 3:43:33 PM com.github.fommil.jni.JniLoader load INFO: already loaded netlib-native_ref-linux-x86_64.so com.github.fommil.netlib.NativeRefLAPACK 

So everything seems to be fine here. But a spark cannot detect libraries. I added this java dependency to my jar assembly

 com.github.fommil.netlib:all:1.1.2 

also, if I try to start the spark shell with this package, it does not work.

 spark-shell --packages com.github.fommil.netlib:all:1.1.2 

Answer 1

It looks like your netlib-java implementation loads NativeRefBLAS, but not NativeSystemBLAS. This means that you have included "com.github.fommil.netlib: all" works fine, because without it you will use a non-native implementation of F2J. The problem is that you want to use system BLAS (OpenBLAS), instead of the reference implementation that comes with netlib-java. It's probably just a matter of getting the right shared libraries in a place that is visible to your spark performers.

You said that you linked libblas.so.3, but as described in netlib-java readme , you also need to configure libblas.so, liblapack.so and liblapack.so.3:

 sudo apt-get install libatlas3-base libopenblas-base sudo update-alternatives --config libblas.so sudo update-alternatives --config libblas.so.3 sudo update-alternatives --config liblapack.so sudo update-alternatives --config liblapack.so.3