I have a network embedded in TensorFlow that takes a lot of time to learn and therefore wants to profile it to see which parts are causing a lot of work.
To do this, I follow the instructions here to capture runtime and memory information. My code is as follows:
// define network
loss = ...
train_op = tf.train.AdamOptimizer().minimize(loss, global_step=global_step)
// run forward and backward prop for one batch
run_metadata = tf.RunMetadata()
options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
_,loss,sum = sess.run([train_op,loss,sum], feed_dict=fd, options=options, run_metadata=run_metadata)
writer.add_run_metadata(run_metadata, 'step_%d' % step)
Then I can see the "session" in TensorBoard. However, as soon as I load a session, most of the operations on my chart turn orange, as shown below, and for them there is no information about the runtime or memory:
, " ". , , "" "", . , . , .
TF 1.3 Tesla K40c.