Stateful LSTM with layer attachment (forms do not match)

Question

I am trying to create stateful LSTM with Keras and I don’t understand how to add an implementation layer before starting LSTM. The problem is the meaning stateful. If my network is not smooth, adding an implementation layer is pretty straight forward and works.

Operating in terms of LSTM state without an implementation layer, it looks like this:

model = Sequential()
model.add(LSTM(EMBEDDING_DIM,
               batch_input_shape=(batchSize, longest_sequence, 1),
               return_sequences=True,
               stateful=True))
model.add(TimeDistributed(Dense(maximal_value)))
model.add(Activation('softmax'))
model.compile(...)

When adding an Embedding layer, I move the parameter batch_input_shapeto the Embedding layer, that is, only the first layer should know the shape? Like this:

model = Sequential()
model.add(Embedding(vocabSize+1, EMBEDDING_DIM,batch_input_shape=(batchSize, longest_sequence, 1),))
model.add(LSTM(EMBEDDING_DIM,
               return_sequences=True,
               stateful=True))
model.add(TimeDistributed(Dense(maximal_value)))
model.add(Activation('softmax'))
model.compile(...)

The exception that I know is Exception: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=4

So, I'm stuck here at the moment. What trick combines embedding words in LSTM with state?

Answer 1

batch_input_shape Embedding (batch_size, time_steps), time_steps - LSTM/ , batch_size - .

model = Sequential()
model.add(Embedding(
   input_dim=input_dim, # e.g, 10 if you have 10 words in your vocabulary
   output_dim=embedding_size, # size of the embedded vectors
   input_length=time_steps,
   batch_input_shape=(batch_size,time_steps)
))
model.add(LSTM(
   10, 
   batch_input_shape=(batch_size,time_steps,embedding_size),
   return_sequences=False, 
   stateful=True)
)

LSTM Keras. , gist, LSTM .