Interpolated sampling of points in the image using TensorFlow

Question

An image in grayscale I is presented as a 2D tensor (size W, H) and coordinate tensor C (Dim. None, 2). I want to interpret the C lines as coordinates in I , the sample I in these coordinates, using some interpolation (bilinear will probably be good for my use case) and save the received values ​​in a new Tensor P (dimension None, i.e. 1-dimensional with as many entries as C has lines).

Is this possible (effective) with TensorFlow? All I can find are functions for resizing (equidistant oversampling, if you like) images. But I can’t find anything out of the box to select from the coordinate list.

those. I would expect to find something like the tf.interpolate () function:

I = tf.placeholder("float", shape=[128, 128]) C = tf.placeholder("float", shape=[None, 2]) P = tf.interpolate(I, C, axis=[0, 1], method="linear") 

Ideally, I would look for a solution that would allow me to interpolate I in the N dimensional tensor I along dimensions M using C with the shape (None, M) and produce the output signal NM + 1, as indicated by the "axis" parameter in the above code.

(The "image" in my application is not a btw. Picture, it displays data from a physical model (when used as a placeholder) or an alternative model studied (when used as a variable). Now this physical model is 2 degrees of freedom, thus interpolating into " image "is enough for now, but I could take a look at higher dimensional models in the future.)

In case something like this is impossible with existing TensorFlow functions: where do I start when I would like to implement something like this tf.interpolate () operator? (documentation and / or simple code example)

Answer 1

There is no built-in op that does this interpolation, but you have to do this using the composition of existing TensorFlow operations. I would suggest the following strategy for the bilinear case:

• From your tensor C indices, calculate integer tensors corresponding to four corner points. For example (with names suggesting that the beginning is in the upper left corner):

   top_left = tf.cast(tf.floor(C), tf.int32) top_right = tf.cast( tf.concat(1, [tf.floor(C[:, 0:1]), tf.ceil(C[:, 1:2])]), tf.int32) bottom_left = tf.cast( tf.concat(1, [tf.ceil(C[:, 0:1]), tf.floor(C[:, 1:2])]), tf.int32) bottom_right = tf.cast(tf.ceil(C), tf.int32) 

• From each tensor representing a specific corner point, extract the vector of values ​​from I at these points. For example, for the following function, this is for the 2-D case:

   def get_values_at_coordinates(input, coordinates): input_as_vector = tf.reshape(input, [-1]) coordinates_as_indices = (coordinates[:, 0] * tf.shape(input)[1]) + coordinates[:, 1] return tf.gather(input_as_vector, coordinates_as_indices) values_at_top_left = get_values_at_coordinates(I, top_left) values_at_top_right = get_values_at_coordinates(I, top_right) values_at_bottom_left = get_values_at_coordinates(I, bottom_left) values_at_bottom_right = get_values_at_coordinates(I, bottom_right) 

• First calculate the interpolation in the horizontal direction:

   # Varies between 0.0 and 1.0. horizontal_offset = C[:, 0] - tf.cast(top_left[:, 0], tf.float32) horizontal_interpolated_top = ( ((1.0 - horizontal_offset) * values_at_top_left) + (horizontal_offset * values_at_top_right)) horizontal_interpolated_bottom = ( ((1.0 - horizontal_offset) * values_at_bottom_left) + (horizontal_offset * values_at_bottom_right)) 

• Now calculate the interpolation in the vertical direction:

   vertical_offset = C[:, 1] - tf.cast(top_left[:, 1], tf.float32) interpolated_result = ( ((1.0 - vertical_offset) * horizontal_interpolated_top) + (vertical_offset * horizontal_interpolated_bottom)) 

Answer 2

This turned out to be difficult for the closest neighbor, given that TF does not yet have Numpy trim common ( github issue # 206 ), and the fact that gather only works in the first dimension. But here you can get around this using cross-> transpose-> gather-> extract diagonal

 def identity_matrix(n): """Returns nxn identity matrix.""" # note, if n is a constant node, this assert node won't be executed, # this error will be caught during shape analysis assert_op = tf.Assert(tf.greater(n, 0), ["Matrix size must be positive"]) with tf.control_dependencies([assert_op]): ones = tf.fill(n, 1) diag = tf.diag(ones) return diag def extract_diagonal(tensor): """Extract diagonal of a square matrix.""" shape = tf.shape(tensor) n = shape[0] assert_op = tf.Assert(tf.equal(shape[0], shape[1]), ["Can't get diagonal of " "a non-square matrix"]) with tf.control_dependencies([assert_op]): return tf.reduce_sum(tf.mul(tensor, identity_matrix(n)), [0]) # create sample matrix size=4 I0=np.zeros((size,size), dtype=np.int32) for i in range(size): for j in range(size): I0[i, j] = 10*i+j I = tf.placeholder(dtype=np.int32, shape=(size,size)) C = tf.placeholder(np.int32, shape=[None, 2]) C0 = np.array([[0, 1], [1, 2], [2, 3]]) row_indices = C[:, 0] col_indices = C[:, 1] # since gather only supports dim0, have to transpose I1 = tf.gather(I, row_indices) I2 = tf.gather(tf.transpose(I1), col_indices) I3 = extract_diagonal(tf.transpose(I2)) sess = create_session() print sess.run([I3], feed_dict={I:I0, C:C0}) 

So, starting with such a matrix:

 array([[ 0, 1, 2, 3], [10, 11, 12, 13], [20, 21, 22, 23], [30, 31, 32, 33]], dtype=int32) 

This code retrieves the diagonal above the main

 [array([ 1, 12, 23], dtype=int32)] 

There some kind of magic happens with the operators [], which turn into Squeeze and Slice