How to reuse an existing neural network to learn new with TensorFlow?

Question

I want to train a new neural network using TensorFlow, reusing the lower levels of an existing neural network (which is already trained). I want to reset the top layers of the existing network and replace them with new layers, and I also want to block the bottom layers to prevent them from being redirected. Here's some ascii art to summarize this:

*Original model*           *New model*

Output Layer                Output Layer (new)
     |                          |
Hidden Layer 3             Hidden Layer 3 (copied)
     |             ==>          |
Hidden Layer 2             Hidden Layer 2 (copied+locked)
     |                          |
Hidden Layer 1             Hidden Layer 1 (copied+locked)
     |                          |
   Inputs                     Inputs

What a good way to do this?

Edit

My initial network was created as follows:

X = tf.placeholder(tf.float32, shape=(None, 500), name="X")
y = tf.placeholder(tf.int64, shape=(None), name="y")

hidden1 = fully_connected(X, 300, scope="hidden1")
hidden2 = fully_connected(hidden1, 100, scope="hidden2")
hidden3 = fully_connected(hidden2, 50, scope="hidden3")
output = fully_connected(hidden3, 5, activation_fn=None, scope="output)

xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits, y)
loss = tf.reduce_mean(xentropy, name="loss")
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)
training_op = optimizer.minimize(loss)

init = tf.initialize_all_variables()
saver = tf.train.Saver()

# ... Train then save the network using the saver

What code will load this network, block the 2 lower hidden layers and replace the output level? If possible, it would be great to cache the output of the upper locked level (hidden2) for each input to speed up learning.

additional information

retrain.py How-To ( ). , (.. ) . , . + : . , , (.. ) (, 3 ).

!

Answer 1

TensorFlow (Variable s), . , , . . , Variable W1, b1, W2, b2, W3, b3, Woutput boutput. , Optimizer , loss - , , , :

opt = GradientDescentOptimizer(learning_rate=0.1)
grads_and_vars = opt.compute_gradients(loss, var_list=[W3, b3, Woutput, boutput])
train_op = opt.apply_gradients(grads_and_vars)

: opt.minimize(loss, var_list) , , .

opt.compute_gradients , , . , 3 . , , tf.assign .