Keras with Tensorflow GPU support. MKL ERROR: parameter 4 was invalid when entering DLASCL

Question

I installed Tensorflow with GPU and Keras support on Wednesday in Anaconda (v1.6.5) using the following commands:

conda install -n EnvName tensorflow-gpu conda install -n EnvName -c conda-forge keras-gpu 

I have NVIDIA Quadro 2200K on my machine with driver v384.66, cuda-8.0, cudnn 7.0

When I try to run Python code with Keras in the training phase, I get the following

Intel MKL ERROR: parameter 4 was incorrect when entering DLASCL.

and later

File "/home/User/anaconda3/envs/keras_gpu/lib/python3.6/site-packages/numpy/linalg/linalg.py", line 99, in _raise_linalgerror_svd_nonconvergence raise LinAlgError ("SVD does not converge") nump. linalg.LinAlgError: SVD does not converge

Other relevant sources suggest checking the data for NaN and Infs, but my data is clean. By the way, the version of the installation processor is working fine, the problem only occurs when trying to run on the GPU

I tried reinstalling Anaconda, reinstalling CUDA and numpy, but this did not work.

Answer 1

The problem was with the mkl package (2018.0.0) - it looks like it was recently released and conflicts with the version of some packages that come with Tensorflow (1.3.0) and Keras (2.0.5) via conda *.

So, I manually downgraded mkl using Anaconda Navigator to version 11.3.3, which automatically lowered the rating of other packages, and now everything works fine.

Answer 2

If you use Anaconda, this will help:

 conda update mkl