Why is learning loss increasing at the beginning of each era?

Question

I am training a linear regression model. I use tf.contrib.data to prepare the dataset, shuffle it and serve it in batches:

  dataset = tf.contrib.data.TFRecordDataset(filename)
  dataset = dataset.map(
      _parse_function, num_threads=16, output_buffer_size=100 * batch_size)
  dataset = dataset.repeat(5)
  dataset = dataset.shuffle(buffer_size=100000)
  dataset = dataset.padded_batch(batch_size, padded_shapes=([None], [None]))
  iterator = dataset.make_initializable_iterator()
  x_inputs, y_ = iterator.get_next()

The following is our learning loss:

It is very strange that at the beginning of each era (iteration = 100 thousand), we have an impulse in training loss. If the learning process continues, we see the same pattern at the beginning of the following eras.

Answer 1

, 100000 , . , , , , , .

, :

• , buffer_size, Dataset.shuffle(), , . ( , .)

• , - , .

• , , :

  dataset = tf.data.Dataset.list_files(shard_filenames)
  dataset = dataset.shuffle(len(shard_filenames))
  dataset = dataset.flat_map(lambda filename: tf.data.TFRecordDataset(filename))
  # ...
  

• (3), Dataset.interleave() Dataset.flat_map() .

Answer 2

: https://discuss.pytorch.org/t/strange-behavior-with-sgd-momentum-training/7442

, .