How to check NaN in gradients in Tensorflow when updating?

Question

Everything,

When you are preparing a large model with a large number of samples, some samples may cause a NaN gradient when updating the parameter.

And I want to find these samples. Meanwhile, I do not want this gradient of batch samples to update the model parameter, because this may be the reason that the model parameter NaN.

So who has a good idea to deal with this problem?

My code is as follows:

# Create an optimizer. params = tf.trainable_variables() opt = tf.train.AdamOptimizer(1e-3) gradients = tf.gradients(self.loss, params) max_gradient_norm = 10 clipped_gradients, self.gradient_norms = tf.clip_by_global_norm(gradients, max_gradient_norm) self.optimizer = opt.apply_gradients(zip(clipped_gradients, params)) 

Answer 1

You can check if your NaN gradients have tf.check_numerics :

 grad_check = tf.check_numerics(clipped_gradients) with tf.control_dependencies([grad_check]): self.optimizer = opt.apply_gradients(zip(clipped_gradients, params)) 

grad_check will throw an InvalidArgument if clipped_gradients is NaN or infinity.

tf.control_dependencies verifies that grad_check is evaluated before applying gradients.

Also see tf.add_check_numerics_ops() .

Answer 2

You can use tf.is_nan in combination with tf.cond to execute the rest of your code if the loss is not NaN.