Custom Keras Metric Based on Predicted Values

Question

I would like to implement a custom metric in keras that calculates feedback, assuming the top k% most likely y_pred_probs is true.

In numpy I would do it as follows. Sort y_preds_probs. Then take the value at the k index. Note k=0.5 will give the median value.

 kth_pos = int(k * len(y_pred_probs)) threshold = np.sort(y_pred_probs)[::-1][kth_pos] y_pred = np.asarray([1 if i >= threshold else 0 for i in y_pred_probs]) 

The response from the Keras user-defined decision threshold for accuracy and recall is fairly close, but suggests that the threshold for determining which y_pred is considered true is already known. I would like to combine the approaches and implement a search for the threshold_value value based on k and y_pred in the Keras backend, if possible.

 def recall_at_k(y_true, y_pred): """Recall metric. Computes the recall over the whole batch using threshold_value from k-th percentile. """ ### threshold_value = # calculate value of k-th percentile of y_pred here ### # Adaptation of the "round()" used before to get the predictions. Clipping to make sure that the predicted raw values are between 0 and 1. y_pred = K.cast(K.greater(K.clip(y_pred, 0, 1), threshold_value), K.floatx()) # Compute the number of true positives. Rounding in prevention to make sure we have an integer. true_positives = K.round(K.sum(K.clip(y_true * y_pred, 0, 1))) # Compute the number of positive targets. possible_positives = K.sum(K.clip(y_true, 0, 1)) recall_ratio = true_positives / (possible_positives + K.epsilon()) return recall_ratio 

Answer 1

Thanks for quoting my previous answer.

In this case, if you use the shadoworflow backend, I suggest you use this shadoworflow function:

 tf.nn.in_top_k( predictions, targets, k, name=None ) 

It outputs the tensor bools, 1 if the answer belongs to the vertex k and 0 if not.

If you need more information, I linked the documentation with tensor flow. I hope this helps. :-)