Why isotify plays events?

Question

I need to process large (~ 100) syslog messages using Perl and Linux :: Inotify2 .

I wrote a test script that continuously generates log messages. For event handling, my Perl script looks like this:

#!/usr/bin/perl use Linux::Inotify2 ; use Time::HiRes qw(usleep nanosleep); # create a new object my $inotify = new Linux::Inotify2 or die "Unable to create new inotify object: $!" ; # create watch $inotify->watch ("/var/log/messages", IN_ACCESS|IN_OPEN|IN_CLOSE|IN_MODIFY|IN_Q_OVERFLOW) or die "watch creation failed" ; my $count=0; while () { my @events = $inotify->read; unless (@events > 0) { print "read error: $!"; last ; } #printf "mask\t%d\n", $_->mask foreach @events ; $count++; print $count."\n"; usleep(100000); } 

If I do not comment on the usleep function to simulate processing, I notice that when I stop the script log generator, the inotify script does not catch up with it. In other words, the inotify Perl script loses events.

I also do not see overflow messages.

How to make sure that even if my processing is slow, I do not lose the message. In other words, how to define a “buffer” where messages can be temporarily stored?

Answer 1

I will try to solve your problem with the root, although this does not exactly answer your question.

I think the use of inotify is too low level to allow this. If I were you, I would look at a higher-level solution that frees you from the details of searching for new events and allows you to focus on processing the events of the journal. No need to reinvent the wheel when there are already a bunch of good ones.

My first choice would be to use the modern syslog daemon (I personally prefer Syslog-NG, but rsyslog will work quite well too) and bind it directly to your script. Instead of having your script look at all the tracking work as new events arrive, your script simply processes stdin , and the syslog daemon automatically sends new logs to your script as they arrive. This method has been used successfully many times.

My second choice would be to let the higher-level Perl module do as much hard work as possible. The first thing I would like to find would be File::Tail . As before, this saves you the trouble of viewing the file itself and allows you to focus on processing.

Answer 2

Looking at Inotify on CPAN and you tried it with a callback to check all errors:

  $inotify->watch ("/etc/passwd", IN_ACCESS | IN_MODIFY, sub { my $e = shift; print "$e->{w}{name} was accessed\n" if $e->IN_ACCESS; print "$e->{w}{name} was modified\n" if $e->IN_MODIFY; print "$e->{w}{name} is no longer mounted\n" if $e->IN_UNMOUNT; print "events for $e->{w}{name} have been lost\n" if $e->IN_Q_OVERFLOW; }); 

In addition, you look at almost every event in the file. I watched only the modification events or a smaller subset of events to find out if the problem exists.