How do you manage logging performance?

Question

We have a messaging system where low latency is crucial. I recently discovered that although we maintain a high level in our system, we see some "outliers." (Messages that take a lot longer than they should) When we unregister, our systems do not show any of these outliers.

Right now, our logging is basically just a wrapped stream with some logging level features similar to log4j (debug, fatal, debug, etc.).

I was wondering what others are doing to control the effectiveness of logging, especially in message processing? How do you control these related I / O actions? Do you rinse it? Are you moving to databases?

Any recommendations for optimizing logging are welcome.

Note. I understand that in our system there may be other problems causing outliers, but for this issue I am only interested in optimizing logging, thanks.

Also: registration is required for our system.

Answer 1

I assume that OS depends on some degree.

In win32, our logging subsystem simply queues the message queue for the logging thread that processes disk I / O.

This separates disk I / O performance from time-critical threads and gives us good control over how and when the queue is blocked.

Answer 2

Like what Roddy said, we also put the message queue in a thread-safe queue and have a separate thread with a lower priority that does the actual disk I / O.

In the background thread, we also have a limit on the number of messages that can be recorded immediately (dequeued), so for something more than this, we will return the backstream of the background.