Unpredictable CUDNN_STATUS_NOT_INITIALIZED on Windows

Question

I run keras neural network training and forecasting on the GTX 1070 on Windows 10. In most cases it works, but from time to time it complains

E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_dnn.cc:359] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_dnn.cc:366] error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_dnn.cc:326] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
F c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\kernels\conv_ops.cc:659] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms)

It is impossible to explain either the literal meaning of the error or the OOM error.

How to fix?

Answer 1

. Windows 10 Nvidia GEforce 920M. cudnn. CUDA, tenorflow, . CUDA CUDNN. , .

Answer 2

gpu set gpu per_process_gpu_memory_fraction.

, , , .

.7 .

Answer 3

Windows10 Keras. , .

https://github.com/fchollet/keras/issues/1538

import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.3
set_session(tf.Session(config=config))

.

Answer 4

tf doku GPU

-, allow_growth, GPU-, : , , , GPU, GPU TensorFlow. , , . , ConfigProto :

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config, ...)

with tf.Session(graph=graph_node, config=config) as sess:
     ...

The second method is the per_process_gpu_memory_fraction parameter, which determines the fraction of the total memory that each visible GPU should allocate. For example, you can say that TensorFlow allocates 40% of the total memory of each GPU:

config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4
session = tf.Session(config=config, ...)