How to visualize the behavior of many parallel multi-stage processes?

Question

Suppose I have a tone (continuous stream) of processing requests, and each request has several steps. For example: “connecting to a data source”, “reading data from a data source”, “checking data”, “processing data”, “connecting to a data receiver”, “writing the result to a data receiver”.

What visualization methods or even tools are well suited to visualize the behavior of such a system?

I would like to see which stages take a lot of time, and how the stages of different queries are aligned relative to each other (for example, to see that the data source responds longer when many queries are accessed at once).

If there were only a few dozen queries, I would be fine with a few dozen separate color charts, but for a few thousand that don't fit well. I think I can get away with N color graphs, where N is the "concurrency factor", but 1) maybe something is better, 2) maybe there are tools for this?

PS Shameless plugin: as soon as I figure out the best way to render, I will add it to my great tool called timeplot ;)

PPS Another shameless plugin: I decided to write a separate tool: splot . Here's what it can do based on a trivially simple log and single-line awk:

It shows 160 cores of a cluster performing the tasks passed to them by RabbitMQ. Blue - "data sampling", orange - "calculation", white - "does nothing." This chart immediately shows several problems that are very difficult to find just by looking at the logs.

Answer 1

I have multiprocessor software that runs on a machine with 15 cores. That's what I'm doing.

Record all messages in syslog. Finally, write down the (selected) last 20 minutes of the journal data at http://www.simile-widgets.org/timeline . Keep track of what is logged when and templates I use the syslog viewer. You can find what suits you. http://www.google.com/search?aq=0&oq=syslog+vi&sourceid=chrome&ie=UTF-8&q=syslog+viewer

Hope this helps.