Tensorflow embedding_lookup

Question

I am trying to learn the word-representation of the imdb dataset from scratch via the TensorFlow function tf.nn.embedding_lookup() . If I understand it correctly, I have to set the embedding layer in front of another hidden layer, and then, when I perform gradient descent, the layer will “study” the representation of words in the scales of this layer. However, when I try to do this, I get a form error between my injection layer and the first fully connected layer of my network.

 def multilayer_perceptron(_X, _weights, _biases): with tf.device('/cpu:0'), tf.name_scope("embedding"): W = tf.Variable(tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),name="W") embedding_layer = tf.nn.embedding_lookup(W, _X) layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(embedding_layer, _weights['h1']), _biases['b1'])) layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, _weights['h2']), _biases['b2'])) return tf.matmul(layer_2, weights['out']) + biases['out'] x = tf.placeholder(tf.int32, [None, n_input]) y = tf.placeholder(tf.float32, [None, n_classes]) pred = multilayer_perceptron(x, weights, biases) cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred,y)) train_step = tf.train.GradientDescentOptimizer(0.3).minimize(cost) init = tf.initialize_all_variables() 

The error I am getting is:

 ValueError: Shapes TensorShape([Dimension(None), Dimension(300), Dimension(128)]) and TensorShape([Dimension(None), Dimension(None)]) must have the same rank 

Answer 1

The form error occurs because you use the two-dimensional tensor x to index the two-dimensional tensor of the embedding W Think of tf.nn.embedding_lookup() (and its close relative tf.gather() ), taking each integer i in x and replacing it with the string W[i, :] tf.gather() W[i, :] . From the error message, we can conclude that n_input = 300 and embedding_size = 128 . In general, the result of tf.nn.embedding_lookup() number of dimensions equal to rank(x) + rank(W) - 1 & hellip; in this case 3. An error occurs when you try to multiply this result by _weights['h1'] , which is a (two-dimensional) matrix.

To fix this code, it depends on what you are trying to do and why you are passing the input matrix for the attachment. One common task is to combine the embedding vectors for each input example into one line per example, using an operation like tf.reduce_sum() . For example, you can do the following:

 W = tf.Variable( tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0) ,name="W") embedding_layer = tf.nn.embedding_lookup(W, _X) # Reduce along dimension 1 (`n_input`) to get a single vector (row) # per input example. embedding_aggregated = tf.reduce_sum(embedding_layer, [1]) layer_1 = tf.nn.sigmoid(tf.add(tf.matmul( embedding_aggregated, _weights['h1']), _biases['b1'])) 

Answer 2

Another possible solution: instead of adding embedding vectors, combine these vectors into one vector and increase the number of neurons in the hidden layer.
I used:
embedding_aggregated = tf.reshape(embedding_layer, [-1, embedding_size * sequence_length])
In addition, I changed the number of neurons in the hidden layer to embedding_size * sequence_length . Observation. Accuracy also improved when using concatenation rather than adding.