Tiforflow cifar sync point

Question

Reading https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py for the function the average_gradientsfollowing comment Note that this function provides a synchronization point across all towers.is provided: Is the function a average_gradientsblocking call and what does it mean synchronization point?

I assume this is a blocking call, since in order to calculate the average value of the gradients, each gradient must be individually calculated? But where is the lock code waiting for individual gradients to be calculated?

Answer 1

average_gradients . tenorflow, . , , tower_grads, , for.

, , - . -, for for i in xrange(FLAGS.num_gpus) "" . :

→ → tower_grads

"" gpu with tf.device('/gpu:%d' % i), ( ). , tower_grads , , "" . Tensorflow , "", tower_grads, , average_gradients. , sess.run([train_op, loss]), .