Tensorflow: Single Input Multiple GPU Queue

Question

For example, a tenor10 multi-GPU with a strain gauge , it seems (correct me if I am mistaken) that a single queue of training images is created on the GPU. Isn't the “correct” way to do things have a single queue that feeds all the towers? If so, is there access to the shared queue available?

Answer 1

You are correct that the code for the CIFAR-10 model uses several input queues (via several calls to cifar10.distorted_inputs() via cifar10.tower_loss() ).

The easiest way to use a shared queue between GPUs is to do the following:

• Increase the batch size to N, where N is the number of GPUs.

• Move the call to cifar10.distorted_inputs() from cifar10.tower_loss() and out of the loop on the GPU .

• Separate the images and labels tensors that are returned from cifar10.distorted_inputs() along the 0th (batch) dimension:

   images, labels = cifar10.distorted_inputs() split_images = tf.split(0, FLAGS.num_gpus, images) split_labels = tf.split(0, FLAGS.num_gpus, labels) 

• Modify cifar10.tower_loss() to take the arguments to images and labels and call it like this:

   for i in xrange(FLAGS.num_gpus): with tf.device('/gpu:%d' % i): with tf.name_scope('%s_%d' % (cifar10.TOWER_NAME, i)) as scope: loss = tower_loss(scope, split_images[i], split_labels[i])