How to select the last valid output values ​​from the tensor flow RNN

Question

I train the LSTM cell on batches of sequences that have different lengths. tf.nn.rnn has a very convenient sequence_length parameter, but after calling it, I don’t know how to select the output lines corresponding to the last time step of each element in the package.

My code basically looks like this:

 lstm_cell = tf.nn.rnn_cell.LSTMCell(num_lstm_units, input_size) lstm_outputs, state = tf.nn.rnn(lstm_cell, input_list, dtype=tf.float32, sequence_length=sequence_lengths) 

lstm_outputs is a list with LSTM output at each time step. However, each element in my batch has a different length, so I would like to create a tensor containing the last LSTM output valid for each element in my batch.

If I could use numpy indexing, I would just do something like this:

 all_outputs = tf.pack(lstm_outputs) last_outputs = all_outputs[sequence_lengths, tf.range(batch_size), :] 

But it turns out that for starters begin shadoworflow does not support it (I know the function request ).

So how can I get these values?

Answer 1

A more acceptable workaround was posted by danijar on the function request page that I linked to in the question. There is no need to evaluate tensors, which is a big plus.

I got it to work with tensor flow 0.8. Here is the code:

 def extract_last_relevant(outputs, length): """ Args: outputs: [Tensor(batch_size, output_neurons)]: A list containing the output activations of each in the batch for each time step as returned by tensorflow.models.rnn.rnn. length: Tensor(batch_size): The used sequence length of each example in the batch with all later time steps being zeros. Should be of type tf.int32. Returns: Tensor(batch_size, output_neurons): The last relevant output activation for each example in the batch. """ output = tf.transpose(tf.pack(outputs), perm=[1, 0, 2]) # Query shape. batch_size = tf.shape(output)[0] max_length = int(output.get_shape()[1]) num_neurons = int(output.get_shape()[2]) # Index into flattened array as a workaround. index = tf.range(0, batch_size) * max_length + (length - 1) flat = tf.reshape(output, [-1, num_neurons]) relevant = tf.gather(flat, index) return relevant 

Answer 2

This is not a pleasant solution, but can you evaluate your results and then just use numpy indexing to get the results and create a tensor variable from this? This can work as a stop gap until the tensor flow gets this function. eg.

 all_outputs = session.run(lstm_outputs, feed_dict={'your inputs'}) last_outputs = all_outputs[sequence_lengths, tf.range(batch_size), :] use_this_as_an_input_to_new_tensorflow_op = tf.constant(last_outputs) 

Answer 3

if you are only interested in the last valid output, you can get it through the state returned by tf.nn.rnn() , assuming that it is always a tuple (c, h), where c is the last state and h is the last output. When the state is LSTMStateTuple , you can use the following code snippet (working in tensorflow 0.12):

 lstm_cell = tf.nn.rnn_cell.LSTMCell(num_lstm_units, input_size) lstm_outputs, state = tf.nn.rnn(lstm_cell, input_list, dtype=tf.float32, sequence_length=sequence_lengths) last_output = state[1]