I am using tenorflow 1.1 inside nvidia docker.
I'm currently trying to run two different convolutional neural networks that output in two separate threads, accessing the same gpu in a python process.
First, I upload two different models:
Model 1:
with self.sess.as_default():
saver = tf.train.import_meta_graph('saved_models/cnn_model112.ckpt.meta')
saver.restore(self.sess, 'saved_models/cnn_model112.ckpt')
self.graph = self.sess.graph
with self.graph.as_default():
self.sess.run(tf.global_variables_initializer())
Model 2:
with tf.gfile.GFile(frozen_graph_filename, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
with tf.Graph().as_default() as g1:
tf.import_graph_def(
graph_def,
input_map=None,
return_elements=None,
name=prefix,
op_dict=None,
producer_op_list=None
)
self.sess = tf.Session(graph=self.graph)
These two sessions live in separate facilities. Each object spins a thread to perform output from their associated model and is aimed at this function:
def run(self, stop_event, streams, parent_videos, output, netType):
if self.sess:
with self.sess.as_default():
while not stop_event.is_set():
ret, frame = stream.readSingleFrame()
if ret and frame is not None:
self.sess.run(self.predict, feed_dict={ self.input: [frame]})
This results in the following error:
2017-07-13 09:28:17.346614: E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS
2017-07-13 09:28:17.346866: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:203] Unexpected Event status: 1
, , , , session.run(), . , . , .
, GPU, , tenorflow gpu.
, , ?