I am developing a distributed system where the node wizard runs a bunch of work nodes on remote machines. Since I use Python and want to take advantage of the fact that each physical machine has several cores, I want to run several working nodes on the machine (GIL, etc.). In addition, each working node can vary greatly in the size of the processor that each "cycle" requires. However, I can divide the work nodes into several parts, and my initial strategy would be to create more work nodes than on each machine. The logic is that if several nodes require more CPUs, they can occupy the kernel for a longer time. (If each node was already connected to the CPU, it could not suddenly require more CPU.)
This leads me to the question: How can I accurately measure the processor time of a python process?
I can’t measure time naively, I need time spent specifically for this process. That is, for each process, I want the number X, which, as far as possible, represents the amount of CPU resources spent exclusively on this process, regardless of unrelated processes. (I looked at Python getrusage, but it seems to only give 2 decimal precision points on ubuntu, which is not enough. EDIT: This also happens if I use getrusage () directly in C, at most 0.01 seconds. No cigar)
My specific use case would be to measure the processor time of each node loop, from start to finish, where End occurs approximately 0-30 ms after startup.
The best answer is a portable way to do this in Python. Techniques requiring the use of the C extension are excellent.
source share