TensorFlow: how to use CudnnLSTM with variable input length (e.g. dynamic_rnn)?

Question

I would like to speed up my LSTM network, but since I use it for OCR (where the sequences are of variable length), I cannot use a simple LSTM implementation. That is why I use "tf.nn.dynamic_rnn".

It is based on the reference RNN in the tensor stream ( https://github.com/tensorflow/tensorflow/blob/754048a0453a04a761e112ae5d99c149eb9910dd/tensorflow/contrib/cudnn_rnn/python/kernel_tests/cudnn.n it doesnn.nudnr.nudnr like others). I suppose it may not be possible to use variable-length CUDNNs, but maybe someone will succeed in doing this?

Secondly, this is the use of "tf.nn.bidirectional_dynamic_rnn", since I would like to use Bi-LSTM for text recognition. But this should be resolved after the implementation of the first part.

Change: It seems that "tf.contrib.cudnn_rnn.CudnnLSTM" has a "bidirectional" implementation inside. Thus, the only unknown is that CUDNN can be used with a variable input sequence.

Or any working example using CudnnLSTM may be useful.

Answer 1

Just found this:

Currently tf.contrib.cudnn_rnn.CudnnLSTM does not support packets with sequences of different lengths, so this is usually not an option for use.

: http://returnn.readthedocs.io/en/latest/tf_lstm_benchmark.html

Answer 2

CuDNNLSTM .

from random import random
from numpy import array
from numpy import cumsum
from keras.models import Sequential
from keras.layers import CuDNNLSTM
from keras.layers import Dense
from keras.layers import TimeDistributed

# create a sequence classification instance
def get_sequence(n_timesteps):
    # create a sequence of random numbers in [0,1]
    X = array([random() for _ in range(n_timesteps)])
    # calculate cut-off value to change class values
    limit = n_timesteps/4.0
    # determine the class outcome for each item in cumulative sequence
    y = array([0 if x < limit else 1 for x in cumsum(X)])
    # reshape input and output data to be suitable for LSTMs
    X = X.reshape(1, n_timesteps, 1)
    y = y.reshape(1, n_timesteps, 1)
    return X, y

# define problem properties
n_timesteps = 20

# define CuDNNLSTM
model = Sequential()
model.add(CuDNNLSTM(20, input_shape=(n_timesteps, 1), return_sequences=True))
model.add(TimeDistributed(Dense(1, activation='sigmoid')))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['acc'])

# train CuDNNLSTM
for epoch in range(10):
    # generate new random sequence
    X,y = get_sequence(n_timesteps)
    # fit model for one epoch on this sequence
    model.fit(X, y, epochs=1, batch_size=1, verbose=2)

# evaluate CuDNNLSTM
X,y = get_sequence(n_timesteps)
yhat = model.predict_classes(X, verbose=0)
for i in range(n_timesteps):
    print('Expected:', y[0, i], 'Predicted', yhat[0, i])

:

Epoch 1/1
- 0s - loss: 0.6943 - acc: 0.4000
Epoch 1/1
- 0s - loss: 0.6830 - acc: 0.4500
Epoch 1/1
- 0s - loss: 0.7007 - acc: 0.3500
Epoch 1/1
- 0s - loss: 0.6893 - acc: 0.4500
Epoch 1/1
- 0s - loss: 0.6764 - acc: 0.5000
Epoch 1/1
- 0s - loss: 0.6890 - acc: 0.5000
Epoch 1/1
- 0s - loss: 0.6612 - acc: 0.6000
Epoch 1/1
- 0s - loss: 0.6621 - acc: 0.6000
Epoch 1/1
- 0s - loss: 0.6736 - acc: 0.6000
Epoch 1/1
- 0s - loss: 0.6630 - acc: 0.6000

:

Expected: [0] Predicted [0]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [0] Predicted [1]
Expected: [1] Predicted [1]
Expected: [1] Predicted [1]
Expected: [1] Predicted [1]
Expected: [1] Predicted [1]
Expected: [1] Predicted [1]
Expected: [1] Predicted [1]
Expected: [1] Predicted [1]

Answer 3

, TensorFlow 1.13 : https://github.com/tensorflow/tensorflow/blob/2f672ee9562a452f8dbfa259a8ccec56367e9b17/tensorflow/contrib/cudnn_rnn/python/layers/cudnn_r.

, tf-nightly-gpu sequence_lengths=lengths lenghts - tf.int32 [batch_size], .