Cross-entropy calculation in TensorFlow

Question

It is difficult for me to calculate cross-entropy in a tensor flow. In particular, I use the function:

tf.nn.softmax_cross_entropy_with_logits() 

Using apparently simple code, I can get it to return zero

 import tensorflow as tf import numpy as np sess = tf.InteractiveSession() a = tf.placeholder(tf.float32, shape =[None, 1]) b = tf.placeholder(tf.float32, shape = [None, 1]) sess.run(tf.global_variables_initializer()) c = tf.nn.softmax_cross_entropy_with_logits( logits=b, labels=a ).eval(feed_dict={b:np.array([[0.45]]), a:np.array([[0.2]])}) print c 

returns

 0 

My understanding of cross entropy is as follows:

 H(p,q) = p(x)*log(q(x)) 

Where p (x) is the true probability of the event x and q (x) is the predicted probability of the event x.

There, if any two numbers for p (x) and q (x) are introduced, such that

 0<p(x)<1 AND 0<q(x)<1 

there must be non-zero cross entropy. I expect to use shadoworflow incorrectly. Thanks in advance for any help.

Answer 1

As the saying goes, you cannot pronounce "softmax_cross_entropy_with_logits" without "softmax". Softmax [0.45] - [1] , and log(1) - 0 .

Measures the probability of error in discrete classification problems in which classes are mutually exclusive (each record is in the same class). For example, each CIFAR-10 image is marked with one and only one label: the image may be a dog or a truck, but not both.

NOTE. . Although classes are mutually exclusive, their probabilities are not necessary. All that is required is that each line of labels is equal to a valid probability distribution. If this is not the case, the calculation of the gradient will be incorrect.

If exclusive labels (where one and only one class is true at a time), see sparse_softmax_cross_entropy_with_logits .

WARNING: This op expects unscaled logs, since it executes softmax on logits internally for efficiency. Do not call this op with softmax output, as this will lead to incorrect results.

logits and labels must have the same shape [batch_size, num_classes] and the same type (either float16 , float32 , or float64 ).

Answer 2

In addition to the Don (+1) answer , this answer, written by mrry , may be of interest to you, since it gives a formula for calculating cross-entropy in TensorFlow:

Alternative recording method:

 xent = tf.nn.softmax_cross_entropy_with_logits(logits, labels) 

... will be:

 softmax = tf.nn.softmax(logits) xent = -tf.reduce_sum(labels * tf.log(softmax), 1) 

However, this alternative will be (i) less numerically stable (since softmax can calculate much larger values) and (ii) less efficient (since backprop will over-compute). For real purposes, we recommend using tf.nn.softmax_cross_entropy_with_logits() .