Tensor flow: list of tensors by value

Question

I am trying to work with LSTM in Tensor Flow. I found a tutorial on the Internet where a sequence set was adopted, and the objective function consists of the last LSTM output and known values. However, I would like my objective function to use information from each output. In particular, I am trying to have LSTM learn a lot of sequences (i.e., Learn all the letters in words in a sentence) .:

cell = rnn_cell.BasicLSTMCell(num_units) inputs = [tf.placeholder(tf.float32,shape=[batch_size,input_size]) for _ in range(seq_len)] result = [tf.placeholder(tf.float32, shape=[batch_size,input_size]) for _ in range(seq_len)] W_o = tf.Variable(tf.random_normal([num_units,input_size], stddev=0.01)) b_o = tf.Variable(tf.random_normal([input_size], stddev=0.01)) outputs, states = rnn.rnn(cell, inputs, dtype=tf.float32) losses = [] for i in xrange(len(outputs)): final_transformed_val = tf.matmul(outputs[i],W_o) + b_o losses.append(tf.nn.softmax(final_transformed_val)) cost = tf.reduce_mean(losses) 

This results in an error:

 TypeError: List of Tensors when single Tensor expected 

How do I fix this problem? Does tf.reduce_mean() list of tensor values, or is there some special tensor object that accepts them?

Answer 1

There is a Python list in your losses code. TensorFlow reduce_mean() expects one tensor, not a Python list.

 losses = tf.reshape(tf.concat(1, losses), [-1, size]) 

where size is the number of values ​​you accept softmax should do what you want. See concat ()

But one remark in the code, which seems a little strange, is that you have a list of placeholders for your inputs, while the TensorFlow Tutorial code uses a tensor of order 3 for inputs. Your input is a list of order 2 tensors. I recommend looking at the code in the textbook because it is almost exactly what you are asking.

One of the main files of this tutorial is here . In particular, line 139 is the place where they create their costs. As for your input, lines 90 and 91 are where the input and target placeholders are set. The main conclusion in these two lines is that the entire sequence is transmitted in one placeholder, and not in the list of placeholders.

See line 120 in the ptb_word_lm.py file for where they perform their concatenation.

Answer 2

Working example, check notebook :

 import tensorflow as tf from tensorflow.models.rnn import rnn, rnn_cell print(tf.__version__) #> 0.8.0 batch_size = 2 output_size = input_size = 2 seq_len = 10 num_units = 2 cell = rnn_cell.BasicLSTMCell(num_units) inputs = [tf.placeholder(tf.float32, shape=[batch_size,input_size ]) for _ in xrange(seq_len)] result = [tf.placeholder(tf.float32, shape=[batch_size,output_size]) for _ in xrange(seq_len)] W_o = tf.Variable(tf.random_normal([num_units,input_size], stddev=0.01)) b_o = tf.Variable(tf.random_normal([input_size], stddev=0.01)) outputs, states = rnn.rnn(cell, inputs, dtype=tf.float32) losses = [] for i in xrange(seq_len): final_transformed_val = tf.matmul(outputs[i],W_o) + b_o losses.append(tf.squared_difference(result[i],final_transformed_val)) losses = tf.reshape(tf.concat(1, losses), [-1, seq_len]) cost = tf.reduce_mean(losses) 

To see this in action, you can feed the chart in a hacker way:

 import matplotlib.pyplot as plt import numpy as np step = tf.train.AdamOptimizer(learning_rate=0.01).minimize(cost) sess = tf.InteractiveSession() sess.run(tf.initialize_all_variables()) costs = [] # EXAMPLE # Learn cumsum over each sequence in x # | t | 0 | 1 | 2 | 3 | 4 | ...| # |----------|---|---|---|---|----|----| # | x[:,0,0] | 1 | 1 | 1 | 1 | 1 | ...| # | x[:,0,1] | 1 | 1 | 1 | 1 | 1 | ...| # | | | | | | | | # | y[:,0,0] | 1 | 2 | 3 | 4 | 5 | ...| # | y[:,0,1] | 1 | 2 | 3 | 4 | 5 | ...| n_iterations = 300 for _ in xrange(n_iterations): x = np.random.uniform(0,1,[seq_len,batch_size,input_size]) y = np.cumsum(x,axis=0) x_list = {key: value for (key, value) in zip(inputs, x)} y_list = {key: value for (key, value) in zip(result, y)} err,_ = sess.run([cost, step], feed_dict=dict(x_list.items()+y_list.items())) costs.append(err) plt.plot(costs) plt.show() 

As a beginner with tensor flow, I have yet to find a unified way / best way to handle RNN, but as mentioned above, I am sure this is not recommended. Liked your script as a very nice introduction, thanks for the snippets. In addition, wrg is happening before the scan implementation and the convenience of RNN-tuple-friendly , so be careful