TensorFlow: tf.placeholder and tf.Variable - why size is not required?

Question

I am learning TensorFlow from an example: https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression.ipynb

I have a few questions in the code below: When defining placeholders and variables like X, Y, W, and b, why don't we need to specify their size? How would the code allocate memory without knowing the size of these placeholders / variables? Thank you

# tf Graph Input X = tf.placeholder("float") Y = tf.placeholder("float") # Set model weights W = tf.Variable(rng.randn(), name="weight") b = tf.Variable(rng.randn(), name="bias") # Construct a linear model pred = tf.add(tf.mul(X, W), b) 

Answer 1

TensorFlow tf.placeholder() tensors do not require specifying a form to allow you to combine tensors of different shapes in the later tf.Session.run() . By default, a placeholder has a completely unlimited shape, but you can limit it by passing an optional shape argument. For instance:

 w = tf.placeholder(tf.float32) # Unconstrained shape x = tf.placeholder(tf.float32, shape=[None, None]) # Matrix of unconstrained size y = tf.placeholder(tf.float32, shape=[None, 32]) # Matrix with 32 columns z = tf.placeholder(tf.float32, shape=[128, 32]) # 128x32-element matrix 

When you create a placeholder, TensorFlow does not allocate any memory. Instead, when filing a placeholder, when tf.Session.run() is called, TensorFlow will allocate memory of the appropriate size for the input (and then for any necessary intermediate) tensors.

Note that tf.Variable objects usually require a form when they are created, and this form is output from the first argument to the initializer. In your program, rng.randn() (alias numpy.random.randn() ) returns a scalar value, so the W variables and b will have a scalar shape.

Although the placeholders for your code ( X and Y ) have an unlimited form, some of the operators, such as tf.add() and tf.mul() , have additional requirements for the form of their arguments (namely, that they are compatible with the NumPy Broadcast Rules ) . Since TensorFlow does not know when you plot what the actual shapes of these tensors are, he hopes the user knows what they are doing and performs the check dynamically (during the call to tf.Session.run() ). If instead you restrict the placeholder forms, you let TensorFlow do some validation earlier, and this can help reduce errors.