How to use the "group_by_window" function in TensorFlow

Question

In TensorFlow a new set of pipeline input functions, it is possible to group record sets together using the group_by_window function. It is described in the documentation:

https://www.tensorflow.org/api_docs/python/tf/contrib/data/Dataset#group_by_window

I do not fully understand the explanation here, which is used to describe the function, and I try best to learn by example. I can not find any sample code on the Internet for this function. Could someone please crack a barebones and runnable example of this function to show how it works and what to give this function?

Answer 1

Here is a quick example that I could come up with:

import tensorflow as tf import numpy as np components = np.arange(100).astype(np.int64) dataset = tf.contrib.data.Dataset.from_tensor_slices(components) dataset = dataset.group_by_window(key_func=lambda x: x%2, reduce_func=lambda _, els: els.batch(10), window_size=100) iterator = dataset.make_one_shot_iterator() features = iterator.get_next() sess = tf.Session() sess.run(features) # array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18], dtype=int64) 

The first argument to key_func maps each item in the dataset to a key.

window_size determines the size of the bucket that is assigned to reduce_fund .

In reduce_func you get a block of window_size elements. You can shuffle, set a package or folder, but you want to.

EDIT for dynamic filling and balancing using the group_by_window function more details here :

If you have tf.contrib.dataset that contains (sequence, sequence_length, label) , and the sequence is the tf.int64 tensor:

 def bucketing_fn(sequence_length, buckets): """Given a sequence_length returns a bucket id""" t = tf.clip_by_value(buckets, 0, sequence_length) return tf.argmax(t) def reduc_fn(key, elements, window_size): """Receives `window_size` elements""" return elements.shuffle(window_size, seed=0) # Create buckets from 0 to 500 with an increment of 15 -> [0, 15, 30, ... , 500] buckets = [tf.constant(num, dtype=tf.int64) for num in range(0, 500, 15) window_size = 1000 # Bucketing dataset = dataset.group_by_window( lambda x, y, z: bucketing_fn(x, buckets), lambda key, x: reduc_fn(key, x, window_size), window_size) # You could pad it in the reduc_func, but I'll do it here for clarity # The last element of the dataset is the dynamic sentences. By giving it tf.Dimension(None) it will pad the sencentences (with 0) according to the longest sentence. dataset = dataset.padded_batch(batch_size, padded_shapes=( tf.TensorShape([]), tf.TensorShape([]), tf.Dimension(None))) dataset = dataset.repeat(num_epochs) iterator = dataset.make_one_shot_iterator() features = iterator.get_next() 

Answer 2

I found a TensorFlow test code that contains several uses of the function, but it really is not particularly friendly to beginners.

test cases