TensorFlow: how to measure how much GPU memory each tensor occupies?

Question

I am currently implementing YOLO in TensorFlow, and I'm a little surprised how much memory it takes. On my GPU, I can train YOLO with Darknet with a batch size of 64. On TensorFlow, I can only do this with a batch size of 6, 8 I already don't have enough memory. In the testing phase, I can run the 64 batch size without running out of memory.

• I am wondering how can I calculate how much memory is consumed by each tensor? Are all tensors stored by default in the GPU? Can I just calculate the total memory consumption as a form * 32 bits?

• I noticed that since I use momentum, all my tensors also have tensor /Momentum . Could this also use a lot of memory?

• I am increasing my dataset using the distorted_inputs method, very similar to that defined in the CIFAR-10 tutorial . Maybe this part takes up a huge chunk of memory? I believe that Darknet makes modifications to the CPU.

Answer 1

Now that 1258 is closed, you can enable logging in Python by setting the environment variable before importing TensorFlow:

 import os os.environ['TF_CPP_MIN_VLOG_LEVEL']='3' import tensorflow as tf 

As a result of this there will be many entries. You will want to browse the results to find the corresponding rows. For example:

 grep MemoryLogTensorAllocation train.log 

Answer 2

Sorry for the slow reply. Unfortunately, now the only way to set the log level is to modify the tensorflow / core / platform / logging.h file and recompile, for example,

 #define VLOG_IS_ON(lvl) ((lvl) <= 1) 

Disabled error 1258 for more efficient log management.

Posts

MemoryLogTensorOutput written at the end of each execution of Op and indicates the tensors that contain the outputs of Op. It is useful to know these tensors, since memory is not output until the subordinate Op absorbs the tensors, which can be much later on the large graph.

Answer 3

See the description in this ( commit ). Memory allocation is raw information, although a script is required to collect information in an easily readable form.