Caching a computed value as a constant in TensorFlow

Question

Suppose I want to calculate the least squares coefficients in TensorFlow using a closed-shape solution. I usually do it like this

beta_hat = tf.matmul( tf.matmul(tf.matrix_inverse(tf.matmul(tf.transpose(X), X)), tf.transpose(X)), y ) 

Where X and y are TensorFlow entries corresponding to the covariates and target variable, respectively.

If I then wanted to fulfill a prediction, I would do something like

 y_pred = tf.matmul(X, beta_hat) 

If I had to comply,

 sess.run(y_pred, feed_dict={X: data_X}) 

Of course, I would get an error that I did not provide the necessary value for placeholder y . I would like the flexibility to treat beta_hat as constant after I compute it (so I would not need to define a new placeholder for the new covariance matrix for forecasting). One way to achieve this is

 # Make it constant. beta_hat = sess.run(beta_hat, feed_dict={X: data_X, y: data_y}) y_pred = tf.matmul(X, beta_hat) 

I was wondering if there was a more elegant way to view the tensor as a constant, so I did not need to perform a session and get a constant, and also not create a separate placeholder for the input data that will be used for forecasting.

Here is a sample code that demonstrates the circumstance that I am describing.

 import numpy as np import tensorflow as tf n, k = 100, 5 X = tf.placeholder(dtype=tf.float32, shape=[None, k]) y = tf.placeholder(dtype=tf.float32, shape=[None, 1]) beta = np.random.normal(size=(k, )) data_X = np.random.normal(size=(n, k)) data_y = data_X.dot(beta) data_y += np.random.normal(size=data_y.shape) / 3.0 data_y = np.atleast_2d(data_y).T # Convert to 32-bit precision. data_X, data_y = np.float32(data_X), np.float32(data_y) # Compute the least squares solution. beta_hat = tf.matmul( tf.matmul(tf.matrix_inverse(tf.matmul(tf.transpose(X), X)), tf.transpose(X)), y ) # Launch the graph sess = tf.Session() sess.run(tf.initialize_all_variables()) print "True beta: {}".format(beta) print "Est. beta: {}".format( sess.run(beta_hat, feed_dict={X: data_X, y: data_y}).ravel() ) # # This would error. # y_pred = tf.matmul(X, beta_hat) # print "Predictions:" # print sess.run(y_pred, feed_dict={X: data_X}) # Make it constant. beta_hat = sess.run(beta_hat, feed_dict={X: data_X, y: data_y}) # This will no longer error. y_pred = tf.matmul(X, beta_hat) print "Predictions:" print sess.run(y_pred, feed_dict={X: data_X}) 

Answer 1

Perhaps, intuitively, the easiest way to reuse beta_hat as a constant in the next steps would be to assign it tf.Variable :

 n, k = 100, 5 X = tf.placeholder(dtype=tf.float32, shape=[None, k]) y = tf.placeholder(dtype=tf.float32, shape=[None, 1]) beta = np.random.normal(size=(k, )) data_X = np.random.normal(size=(n, k)) data_y = data_X.dot(beta) data_y += np.random.normal(size=data_y.shape) / 3.0 data_y = np.atleast_2d(data_y).T # Convert to 32-bit precision. data_X, data_y = np.float32(data_X), np.float32(data_y) # Compute the least squares solution. beta_hat = tf.matmul( tf.matmul(tf.matrix_inverse(tf.matmul(tf.transpose(X), X)), tf.transpose(X)), y ) beta_hat_cached = tf.Variable(beta_hat) # Launch the graph sess = tf.Session() print "True beta: {}".format(beta) # Run the initializer, which computes `beta_hat` once: sess.run(beta_hat_cached.initializer, feed_dict={X: data_X, y: data_y}) # To access the value of `beta_hat`, "run" the variable to read its contents. print "Est. beta: {}".format(beta_hat_cached.ravel()) # Use the cached version to compute predictions. y_pred = tf.matmul(X, beta_hat_cached) print "Predictions:" print sess.run(y_pred, feed_dict={X: data_X}) 

Answer 2

mrry really introduced an elegant solution. You should think about correctly attributing his answer if this is really what you want.

However, I think this is a good place to clarify what I consider to be a source of confusion regarding placeholders ... This is not necessarily aimed at the person who asked the question, but I believe that this will be relevant for many newcomers who stumble on this question...

---

Alternates should be considered function inputs. So, first, let's see how this works in Python, and then I will show the equivalent form in Tensorflow ...

If I want to have a function that calculates the output based on the different inputs x and y , then I could do it like this:

 def f(x,y): # For example... return x * y 

In particular, I can call this function with different values ​​for x and y :

 f(1,3) = 3 f(1,4) = 4 f(2,3) = 6 f(2,4) = 8 

However, in my particular case, I can have a fixed y value. Therefore, in my case, it makes no sense to pass y as an argument. Instead, I want to bake the y value in a function and just change x . For this, I can just grab the external y value:

 y = 3 def g(x): return x * y 

Now when I call g , y will have a fixed value of 3:

 g(1) = 3 g(2) = 6 

Similarly, if I also know that x fixed, I could capture the external value of x :

 x = 2 def h(): return g(x) 

Now, when I call h , I implicitly call h()=g(2)=f(2,3) .

This is great, but the problem is that every time I call h , it will repeat the multiplication because it is equivalent to calling f(2,3) . Thus, to improve performance, I can evaluate the expression and then have a function that simply returns this precalculated value:

 val = h() def h23(): return val 

How many times do I call h23 , the multiplication is done only once (in the line val = h() ).

Tensorflow has similar concepts.

If you want to have a function in which you can change both inputs, then you must make placeholder objects for both instances and pass the values ​​of the function in the feed dictionary when you run in the session:

 dtype = tf.float64 shape = () x = tf.placeholder( dtype, shape ) y = tf.placeholder( dtype, shape ) fxy = f(x,y) with tf.Session() as sess: print( sess.run( fxy, {x:1,y:3} ) ) print( sess.run( fxy, {x:1,y:4} ) ) print( sess.run( fxy, {x:2,y:3} ) ) print( sess.run( fxy, {x:2,y:4} ) ) 

However, if one of my values ​​does not change, I can directly initialize it as a constant and create a new function with this value “baked into it”:

 y = tf.constant( 3 ) gx = f( x, y ) with tf.Session() as sess: print( sess.run( gx, {x:1} ) ) print( sess.run( gx, {x:2} ) ) 

The key point is that now I do not need to pass the value for y in the feed dictionary. It is constant and is fixed in the gx expression.
Similarly, if x also a constant, I must declare it like this:

 x = tf.constant(2) h = f(x,y) with tf.Session() as sess: print( sess.run( h ) ) 

As you can see, since all of my variables are constant, I don’t need a feed dictionary at all. This is the Tensorflow equivalent of calling a function with no arguments, such as h() .

However, as before, when I call h , you may need to reevaluate the graph every time. Therefore, I have two options.

• I can calculate the result in numpy and then wrap this value with tensorflow constant.
• I can compute the output in a tensor stream, run it in a session to get the numpy value, and then wrap it in a constant.

In the first option, I would do something like this

 fxy = tf.constant( f(2,3) ) 

Now I precomputed the value of the function outside of Tensorflow, and then wrapped that value as a constant to use in other tenorflow functions.

And vice versa, you only consider option 2 if your function uses some complex Tensorflow functions, or if your function takes a lot of time, and you think that it will be faster for the computer in Tensorflow:

 with tf.Session() as sess: fxy = tf.constant( sess.run( h ) ) 

To understand what’s going on here, recall that

 h = f( tf.constant(1), tf.constant(3) ) 

Therefore, I do not need to pass the dict feed. The sess.run( h ) fragment starts this multiplication inside tensorflow and returns it as a Numpy array. Then, finally, I transfer this value with tf.constant to use it in other Tensorflow functions.