Bidirectional LSTM with batch normalization in Keras

Question

I was wondering how to implement biLSTM with batch normalization (BN) in Keras. I know that the BN layer must be between linearity and nonlinearity, i.e. Activation. This is easy to implement using CNN or Dense layers. But how to do it with biLSTM?

Thanks in advance.

Answer 1

If you want to apply BatchNormalization to LSTM line outputs, you can do it like

from keras.models import Sequential
from keras.layers.recurrent import LSTM
from keras.layers.wrappers import Bidirectional
from keras.layers.normalization import BatchNormalization

model = Sequential()
model.add(Bidirectional(LSTM(128, activation=None), input_shape=(256,10)))
model.add(BatchNormalization())

Essentially, you remove the non-linear LSTM activations (but not the gate activations), and then apply BatchNormalization to the output.

BatchNormalization LSTM, , , , Keras.