How to implement weighted cross-entropy loss in a tensor flow using sparse_softmax_cross_entropy_with_logits

Question

I am starting to use tensor flow (coming from Caffe) and I am using loss sparse_softmax_cross_entropy_with_logits . The function accepts labels like 0,1,...C-1 instead of single encodings. Now I want to use weighting depending on the class label; I know this can be done, possibly with matrix multiplication, if I use softmax_cross_entropy_with_logits (one hot encoding), is there a way to do the same with sparse_softmax_cross_entropy_with_logits ?

Answer 1

 import tensorflow as tf import numpy as np np.random.seed(123) sess = tf.InteractiveSession() # let say we have the logits and labels of a batch of size 6 with 5 classes logits = tf.constant(np.random.randint(0, 10, 30).reshape(6, 5), dtype=tf.float32) labels = tf.constant(np.random.randint(0, 5, 6), dtype=tf.int32) # specify some class weightings class_weights = tf.constant([0.3, 0.1, 0.2, 0.3, 0.1]) # specify the weights for each sample in the batch (without having to compute the onehot label matrix) weights = tf.gather(class_weights, labels) # compute the loss tf.losses.sparse_softmax_cross_entropy(labels, logits, weights).eval() 

Answer 2

In particular, for binary classification, there is weighted_cross_entropy_with_logits that calculates the weighted cross-entropy of softmax.

sparse_softmax_cross_entropy_with_logits tied to a highly efficient unweighted operation (see SparseSoftmaxXentWithLogitsOp , which uses SparseXentEigenImpl under the hood), so it doesn’t “connect”.

In the multiclass case, your option either switches to single-line coding or uses the tf.losses.sparse_softmax_cross_entropy loss function in hacker mode, as has already been suggested, where you will have to transfer the scales depending on the labels in the current batch.

Answer 3

The weight of the classes is multiplied by logits, so it works for sparse_softmax_cross_entropy_with_logits. See this solution for the “Losses for binary class classifier with imbalance in tensor flow” function.

As a side note, you can pass the scales directly to sparse_softmax_cross_entropy

 tf.contrib.losses.sparse_softmax_cross_entropy(logits, labels, weight=1.0, scope=None) 

This method is intended for the loss of transverse entropy using

 tf.nn.sparse_softmax_cross_entropy_with_logits. 

Weight acts as a loss factor. If a scalar is specified, then the loss is simply scaled by the given value. If the weight is a size tensor [batch_size], then the weight loss is applied to each corresponding sample.