Can I run two threads with my own Tensorflow session within the same python process?

I am using tenorflow 1.1 inside nvidia docker.

I'm currently trying to run two different convolutional neural networks that output in two separate threads, accessing the same gpu in a python process.

First, I upload two different models:

Model 1:

with self.sess.as_default():
    saver =     tf.train.import_meta_graph('saved_models/cnn_model112.ckpt.meta')
    saver.restore(self.sess, 'saved_models/cnn_model112.ckpt')
    self.graph = self.sess.graph
with self.graph.as_default():
    self.sess.run(tf.global_variables_initializer())

Model 2:

with tf.gfile.GFile(frozen_graph_filename, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())

with tf.Graph().as_default() as g1:
tf.import_graph_def(
        graph_def,
        input_map=None,
        return_elements=None,
        name=prefix,
        op_dict=None,
        producer_op_list=None
)
self.sess = tf.Session(graph=self.graph)

These two sessions live in separate facilities. Each object spins a thread to perform output from their associated model and is aimed at this function:

def run(self, stop_event, streams, parent_videos, output, netType):
    if self.sess:
        with self.sess.as_default():
            while not stop_event.is_set():
                    ret, frame = stream.readSingleFrame()
                    if ret and frame is not None:
                        self.sess.run(self.predict, feed_dict={ self.input: [frame]})

This results in the following error:

2017-07-13 09:28:17.346614: E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS
2017-07-13 09:28:17.346866: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:203] Unexpected Event status: 1

, , , , session.run(), . , . , .

, GPU, , tenorflow gpu.

, , ?

+4

Source: https://habr.com/ru/post/1681354/


All Articles