Tensor flow: losses are reduced, but accuracy is stable

Question

My team coaches CNN at Tensorflow for binary classification of damaged / acceptable parts. We created our code by modifying the cifar10 example code. In my previous experience with neural networks, I always trained until the loss was very close to 0 (well below 1). However, we are now evaluating our model using the test installed during the training (on a separate GPU), and it seems that the accuracy has stopped increasing after 6.7 thousand steps, while the loss is still steadily decreasing after more than 40 thousand steps. Is it due to retraining? Should we expect to see another burst of accuracy as soon as the loss is very close to zero? Exact maximum accuracy is not acceptable. Should we kill him and continue tuning? What do you recommend? Here is our modified code and graphics of the educational process.

https://gist.github.com/justineyster/6226535a8ee3f567e759c2ff2ae3776b

Images of accuracy and loss

Answer 1

Reducing the loss of binary cross-entropy does not mean an increase in accuracy. Consider label 1, predictions of 0.2, 0.4, and 0.6 over time 1, 2, 3, and a classification threshold of 0.5. time stamps 1 and 2 will lead to a decrease in losses, but without increasing accuracy.

Make sure your model has sufficient capacity by overloading training data. If the model overrides training data, avoid retraining using regularization methods such as loss, regulation of L1 and L2, and data increase.

Finally, confirm the validation data and the training data comes from the same distribution.

Answer 2

Here are my suggestions, one of the possible problems is that your network starts to store data, yes, you should increase regularization,

yes, kill him by reducing the loss for training and getting stable accuracy for checking, this means that your network capacity is low (weak model) trying to go deeper.