I am running sample code for LSTM networks that uses the imdb dataset in Keras. The code can be found in the following link.
imdb_lstm.py
My problem is that as the code progresses, the loss in training decreases and the accuracy of the training increases as expected, but the accuracy of the test fluctuates in the interval, and the loss of the verification increases to a high value. I am enclosing the part of the training phase journal below. Even I notice that when the loss of learning is very small (~ 0.01-0.03), sometimes it increases in the next era, and then decreases again. What I mention can be seen in the eras 75-77. But overall, it is decreasing.
I expect that training accuracy always increases to 0.99-1, and training loss always decreases. Moreover, the accuracy of the test should begin with 0.4 and, ultimately, increase to 0.8. If verification accuracy does not improve over epochs, what is the point of expectation in an epoch? In addition, the accuracy of the test is close to 0.81 at the end.
I also tried my own data and came up with the same situation. I processed my data in a similar way. I mean that my workouts, checks and test points are processed in the same logic as in this code example.
In addition, I did not understand how this code represents the entire sentence after receiving exits from LSTM for each word. Does it spend the average or maximum pool, or does it only require the last exit from the LSTM layer before passing it to the logistic regression classifier?
.
Using Theano backend.
Loading data...
25000 train sequences
25000 test sequences
Pad sequences (samples x time)
X_train shape: (25000, 80)
X_test shape: (25000, 80)
Build model...
Train...
Train on 22500 samples, validate on 2500 samples
Epoch 1/100
22500/22500 [==============================] - 236s - loss: 0.5438 - acc: 0.7209 - val_loss: 0.4305 - val_acc: 0.8076
Epoch 2/100
22500/22500 [==============================] - 237s - loss: 0.3843 - acc: 0.8346 - val_loss: 0.3791 - val_acc: 0.8332
Epoch 3/100
22500/22500 [==============================] - 245s - loss: 0.3099 - acc: 0.8716 - val_loss: 0.3736 - val_acc: 0.8440
Epoch 4/100
22500/22500 [==============================] - 243s - loss: 0.2458 - acc: 0.9023 - val_loss: 0.4206 - val_acc: 0.8372
Epoch 5/100
22500/22500 [==============================] - 239s - loss: 0.2120 - acc: 0.9138 - val_loss: 0.3844 - val_acc: 0.8384
....
....
Epoch 75/100
22500/22500 [==============================] - 238s - loss: 0.0134 - acc: 0.9868 - val_loss: 0.9045 - val_acc: 0.8132
Epoch 76/100
22500/22500 [==============================] - 241s - loss: 0.0156 - acc: 0.9845 - val_loss: 0.9078 - val_acc: 0.8211
Epoch 77/100
22500/22500 [==============================] - 235s - loss: 0.0129 - acc: 0.9883 - val_loss: 0.9105 - val_acc: 0.8234