Where is the code for gradient descent?

Question

Carrying out some experiments with TensorFlow, you want to look at the implementation of some functions, to see exactly how this is done, it started with a simple case tf.train.GradientDescentOptimizer . We downloaded zip from the full source code from github, performed some search queries on the source tree, got:

 C:\tensorflow-master\tensorflow\python\training\gradient_descent.py class GradientDescentOptimizer(optimizer.Optimizer): def _apply_dense(self, grad, var): return training_ops.apply_gradient_descent( 

Ok, so apparently the actual code is in apply_gradient_descent , searched for this ... doesn't exist. Only three occurrences in the entire source tree, all of which are used, not definitions.

What about training_ops ? There is a file with the highest name:

 C:\tensorflow-master\tensorflow\python\training\training_ops.py from tensorflow.python.training import gen_training_ops # go/tf-wildcard-import # pylint: disable=wildcard-import from tensorflow.python.training.gen_training_ops import * # pylint: enable=wildcard-import 

... above all the contents of this file. Hm.

I found this file:

 C:\tensorflow-master\tensorflow\python\BUILD tf_gen_op_wrapper_private_py( name = "training_ops_gen", out = "training/gen_training_ops.py", ) 

which seems to confirm that these and other files are object code generated during the build process, but where is the source code from which they are generated?

So this is the moment when I give up and ask for help. Can someone familiar with the TensorFlow code base tell me where the corresponding source code is?

Answer 1

The implementation also goes to the C ++ source code. Here's ApplyGradientDescent Implementation ( core/kernels/training_ops_gpu.cu.cc ):

 template <typename T> struct ApplyGradientDescent<GPUDevice, T> { void operator()(const GPUDevice& d, typename TTypes<T>::Flat var, typename TTypes<T>::ConstScalar lr, typename TTypes<T>::ConstFlat grad) { Eigen::array<typename TTypes<T>::Tensor::Index, 1> bcast; bcast[0] = grad.dimension(0); Eigen::Sizes<1> single; var.device(d) -= lr.reshape(single).broadcast(bcast) * grad; } }; 

CPU implementation here ( core/kernels/training_ops.cc ):

 template <typename T> struct ApplyGradientDescent<CPUDevice, T> { void operator()(const CPUDevice& d, typename TTypes<T>::Flat var, typename TTypes<T>::ConstScalar lr, typename TTypes<T>::ConstFlat grad) { var.device(d) -= grad * lr(); } };