Tensor flow: optimize gradient descent entry

Question

I have a TensorFlow (convolutional neural network) model that I successfully trained using gradient descent (GD) on some input.

Now, in the second step, I would like to provide the input image as initialization and optimize it with a fixed network parameter using GD. The loss function will be different, but this is a detail.

So my main question is how to say the gradient descent algorithm

• stop optimizing network settings
• optimize by input image

The first can probably be done with this constant variable during the optimizer

Do you guys have any ideas for the second point?

I suppose I can recode the gradient descent algorithm myself using the TF gradient function, but my inner feeling tells me that there should be a simpler way that also allows me to use more complex GD variants (Adam, etc.).

Answer 1

No need for your own SDG implementation. TensorFlow provides all the features:

import tensorflow as tf
import numpy as np

# some input
data_pldhr = tf.placeholder(tf.float32)
img_op = tf.get_variable('input_image', [1, 4, 4, 1], dtype=tf.float32, trainable=True)
img_assign = img_op.assign(data_pldhr)

# your starting image
start_value = (np.ones((4, 4), dtype=np.float32) + np.eye(4))[None, :, :, None]


# override variable_getter
def nontrainable_getter(getter, *args, **kwargs):
    kwargs['trainable'] = False
    return getter(*args, **kwargs)


# all variables in this scope are not trainable
with tf.variable_scope('myscope', custom_getter=nontrainable_getter):
    x = tf.layers.dense(img_op, 10)
    y = tf.layers.dense(x, 10)

# the usual stuff
cost_op = tf.losses.mean_squared_error(x, y)
train_op = tf.train.AdamOptimizer(0.1).minimize(cost_op)

# fire up the training process
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    sess.run(img_assign, {data_pldhr: start_value})

    print(sess.run(img_op))
    for i in range(10):
        _, c = sess.run([train_op, cost_op])
        print(c)
    print(sess.run(img_op))

Answer 2

• represents the image as tf.Variablewithtrainable=True
• initialize this variable with the original image (initial assumption)
• recreate the NN graph using TF variables with trainable=Falseand copy the scales from the prepared NN graph usingtf.assign
• calculate loss function
• TF, .

Answer 3

Another alternative is to use ScipyOptimizerInterface , which allows you to use scipy minimizer. This supports limited minimization.

Answer 4

I am looking for a solution to the same problem, but my model is not easy, since I have an LSTM network with cells created using MultiRNNCell, I do not think that it is possible to get weight and clone the network. Is there a workaround so that I can calculate the gradient from the input?