Keras Stateful LSTM train return_seq = true, not a student

Question

Consider this minimal example:

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
import numpy as np
import matplotlib.pyplot as plt


max = 30
step = 0.5
n_steps = int(30/0.5)

x = np.arange(0,max,step)
x = np.cos(x)*(max-x)/max

y = np.roll(x,-1)
y[-1] = x[-1]

shape = (n_steps,1,1)
batch_shape = (1,1,1)

x = x.reshape(shape)
y = y.reshape(shape)

model = Sequential()
model.add(LSTM(50, return_sequences=True, stateful=True, batch_input_shape=batch_shape))
model.add(LSTM(50, return_sequences=True, stateful=True))

model.add(Dense(1))

model.compile(loss='mse', optimizer='rmsprop')

for i in range(1000):
    model.reset_states()
    model.fit(x,y,nb_epoch=1, batch_size=1)
    p = model.predict(x, batch_size=1)
    plt.clf()
    plt.axis([-1,31, -1.1, 1.1])
    plt.plot(x[:, 0, 0], '*')
    plt.plot(y[:,0,0],'o')
    plt.plot(p[:,0,0],'.')
    plt.draw()
    plt.pause(0.001)

---

As stated in the keras API https://keras.io/layers/recurrent/

the last state for each sample in the index i in the batch will be used as the initial state for the sample index i in the next batch

So, I am using batch_size = 1, and I am trying to predict the next value in a decaying cos-function for each time. The prediction or red dots in the figure below should go in green circles for the script to predict it correctly, however it does not converge ... Is there any idea to get it to learn?

Answer 1

model.fit . optimizer reset . - reset_states - - states from fit - , . :

for epoch in range(1000):
    model.reset_states()
    tot_loss = 0
    for batch in range(n_steps):
        batch_loss = model.train_on_batch(x[batch:batch+1], y[batch:batch+1])
        tot_loss+=batch_loss

    print "Loss: " + str(tot_loss/float(n_steps))
    model.reset_states()
    p = model.predict(x, batch_size=1)