How to solve the loss?

Question

Problem

I run the Deep Neural Network in MNIST, where the loss is defined as follows:

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, label))

It seems that the program is working correctly until I get the loss of nano in the 10,000+th mini-bar. Sometimes a program runs correctly until it ends. I think tf.nn.softmax_cross_entropy_with_logits gives me this error. This is strange because the code simply contains mul and add operations.

Possible Solution

Maybe I can use:

 if cost == "nan": optimizer = an empty optimizer else: ... optimizer = real optimizer 

But I can not find type nan . How to check nan variable or not?

How else can I solve this problem?

Answer 1

Test your learning speed. The larger your network, the more options to explore. This means that you also need to reduce your learning speed.

Answer 2

The reason you get NaN is most likely somewhere in your cost function or softmax you are trying to take a journal with zero, which is not a number. But to answer your specific question about detecting NaN, Python has a built-in ability to test NaN in a math module. For instance:

 import math val = float('nan') val if math.isnan(val): print('Detected NaN') import pdb; pdb.set_trace() # Break into debugger to look around 

Answer 3

I found a similar problem here TensorFlow Cross_entropy NaN Task

Thanks to the author user1111929

 tf.nn.softmax_cross_entropy_with_logits => -tf.reduce_sum(y_*tf.log(y_conv)) 

actually a terrible way to calculate cross entropy. In some examples, some classes can be excluded with certainty after some time, resulting in y_conv = 0 for this sample. This is usually not a problem, since you are not interested in it, but in how cross_entropy is written there, it gives 0 * log (0) for this particular sample / class. Therefore, NaN.

Replacing it

 cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv + 1e-10)) 

or

 cross_entropy = -tf.reduce_sum(y_*tf.log(tf.clip_by_value(y_conv,1e-10,1.0))) 

Solved cash problem.

Answer 4

I do not have your code or data. But tf.nn.softmax_cross_entropy_with_logits should be stable with a valid probability distribution (more details here ). I assume that your data does not meet this requirement. A similar problem has also been discussed here. Which would lead you to:

• Implement your own softmax_cross_entropy_with_logits function, for example. try ( source ):

   epsilon = tf.constant(value=0.00001, shape=shape) logits = logits + epsilon softmax = tf.nn.softmax(logits) cross_entropy = -tf.reduce_sum(labels * tf.log(softmax), reduction_indices=[1]) 

• Update your data to have a reliable probability distribution.