How to reuse RNN in TensorFlow

Question

I want to implement a model such as DSSM (deep semantic similarity model).

I want to train one RNN model and use this model to get three hidden vectors for three different inputs and use this hidden vector to calculate the loss function.

I am trying to execute code in variable scope with reuse = None like:

gru_cell = tf.nn.rnn_cell.GRUCell(size) gru_cell = tf.nn.rnn_cell.DropoutWrapper(gru_cell,output_keep_prob=0.5) cell = tf.nn.rnn_cell.MultiRNNCell([gru_cell] * 2, state_is_tuple=True) embedding = tf.get_variable("embedding", [vocab_size, wordvec_size]) inputs = tf.nn.embedding_lookup(embedding, self._input_data) inputs = tf.nn.dropout(inputs, 0.5) with tf.variable_scope("rnn"): _, self._states_2 = rnn_states_2[config.num_layers-1] = tf.nn.dynamic_rnn(cell, inputs, sequence_length=self.lengths, dtype=tf.float32) self._states_1 = rnn_states_1[config.num_layers-1] with tf.variable_scope("rnn", reuse=True): _, rnn_states_2 = tf.nn.dynamic_rnn(cell,inputs,sequence_length=self.lengths,dtype=tf.float32) self._states_2 = rnn_states_2[config.num_layers-1] 

I use the same inputs and reuse the RNN model, but when I print 'self_states_1' and 'self_states_2', the two vectors are different.

I use with tf.variable_scope("rnn", reuse=True): to calculate "rnn_states_2" because I want to use the same RNN model as "rnn_states_1".

But why do I get different hidden vectors with the same inputs and the same model?

Where am I wrong?

Thanks for your reply.

Update: I find that the reason may be "tf.nn.rnn_cell.DropoutWrapper" when I remove the dropdown shell, the hidden vectors are the same when I add a wrapper, these vectors become different.

So a new question:

How to fix the part of the vector that will “drop out”? By setting the "seed" parameter?

When learning DSSM, should the shutdown action be fixed?

Answer 1

If you structure your code to use tf.contrib.rnn.DropoutWrapper , you can set variational_recurrent=True to your wrapper, which causes the same drop mask to be used at all stages, i.e. the drop mask will be constant. Is this what you want?

Setting the seed parameter to tf.nn.dropout allows you to make sure that you get the same sequence of drop masks each time you run this seed. This does not mean that the screening mask will be constant, you just always see the same rejection mask at a particular iteration. The mask will be different for each iteration.