Is there a way to get very accurate information (number of bytes used, number of CPU instructions) about the process on Unix?

Question

Here is the problem: there is a set of programs written in different languages ​​(mainly Perl and Python). Each program x reads lines from stdin , does some work (analyzes the line, updates data structures, long database queries or fancy network connections, even an IO drive is rare) and possibly prints something to stdout . The task is to write a program f that is given x and stdin , will try lines that would be most difficult to compute for x . The idea is to use such strings for testing and testing x in the future.

Here's what I'm stuck in: f wraps x , reads one line l from stdin , x ready to process l , f goes l to x and immideatly starts collecting staticts about x . The fact is that I can’t find any metric that differs in computational hard and light lines. At the moment, I have tried two approaches:

• Dump all /proc/[x pid]/stat between x runs. It almost does not change (even the CPU ticks).
• Just check the status of x (using the same /proc/[x pid]/stat ) and try measuring the time it starts. It is no different between the lines.

Perhaps there are some indicators of high accuracy? Like the number of processor instructions, or the number of bytes in used memory?

Here is the real Python code I wrote, it is full of details, so this is the last thing to read. I think https://gist.github.com/alexanderkuk/5630079#file-f-py .

Answer 1

You have a lot of problems with your code. The first is:

 def command_is_running(pid): with open('/proc/%d/stat' % pid) as stat: stats = stat.read() return ' R ' in stats def wait_command_processes_line(pid): # stats = ... while command_is_running(pid): # stats = update_stats(stats, pid) return stats 

- employment cycle. It will consume as much CPU as it can, showing .../stat until R disappears. It is not recommended to work with an additional processor when you are trying to get the exact CPU time.

I don’t know how to put the process to sleep until the state of another process changes, so I can’t offer an effective replacement for the busy cycle. But this does not matter because of the second problem: the state of the process is not as predictable as you want.

You made the assumption that the process will become executable at the moment when you write some data in its pipe, and will remain on while working throughout the process of processing this input. It would be very difficult to guarantee that this is true. You said that the "IO drive is rare", but you will need to do better than this and completely eliminate it, including page errors. This is complicated, and you probably didn’t. Therefore, I think that your problem is not that /proc/PID/stat contains incorrect information, but that you are reading it at the wrong time.

You can get around the disk I / O problem by treating state D in the same way as R But he still looks stupid.

Instead of looking at the health of the process, you should find a better indicator that the child process has finished processing the very last line of input. You said that "maybe it prints something in stdout". If you can arrange for him to always print something in stdout for each line of input, then the parent process can wait for this output and an example of using the child processor when it appears.

If you cannot force the child process to provide an external completion indication for each input line, an alternative would be to look at it with the input line when it tries to read the next line of input. Basically, you will use ptrace to implement a specialized strace utility that writes read time on the input channel, writing a string to the pipe only after your trace tells you what it is trying to read.

Perhaps you can even do this with strace and some smart shell scripts.

Another option for this idea would be to use gdb to set a breakpoint in the child process at the beginning of its input processing cycle, and set up a script that will execute every time a breakpoint is hit. The script will collect time information, write the next line to the pipe, and then execute gdb cont .