How to write a common log parser

Question

We need to analyze several log files and run some statistics on the found log entries (such as the number of occurrences of certain messages, spikes in occurrences, etc.). The problem is writing a log parser that processes several log formats and will allow me to add a new log format with very little work.

To simplify the task, I only look at the logs, which basically will look something like this:

[11/17/11 14:07:14:030 EST] MyXmlParser     E   Premature end of file

therefore, each log entry will contain timestamp, originator(log messages), leveland log message. One important detail is that a message can have more than one line (e.g. stacktrace). Another instance of a log entry may be:

17-11-2011 14:07:14 ERROR    MyXmlParser   - Premature end of file

I am looking for a good way to specify the log format, as well as the most suitable technology for implementing the parser. Although I speak of regular expressions, I think it will be difficult to deal with situations such as multi-line messages (e.g. stacktrace).

In fact, the task of writing a parser for a specific log format does not sound so easy when I consider the possibility of multi-line messages. How are you going to parse these files?

Ideally, I could specify something like this as the log format:

[%TIMESTAMP] %ORIGIN %LEVEL %MESSAGE

or

%TIMESTAMP %LEVEL %ORIGIN - %MESSAGE

Obviously, I would have to assign the correct converter for each field so that it would handle it correctly (for example, a timestamp).

Can someone give me some good ideas on how to implement this in a reliable and modular way (I use Java)?

Answer 1

logstash.

Answer 2

AWStats - , , , .

Answer 3

, . , , :

private static final Pattern LINE_PATTERN = Pattern.compile(
  "(\\S+:)?(\\S+? \\S+?) \\S+? DEBUG \\S+? - DEMANDE_ID=(\\d+?) - listener (\\S+?) : (\\S+?)");

public static EventLog parse(String line) throws ParseException {
    String demandId;
    String listenerClass;
    long startTime;
    long endTime;

    SimpleDateFormat sdf = new SimpleDateFormat(DATE_PATTERN);
    Matcher matcher = LINE_PATTERN.matcher(line);
    if (matcher.matches()) {
        int offset = matcher.groupCount()-4; // 4 interesting groups, the first is optional
        demandeId = matcher.group(2+offset);
        listenerClass = matcher.group(3+offset);
        long time = sdf.parse(matcher.group(1+offset)).getTime();
        if ("starting".equals(matcher.group(4+offset))) {
            startTime = time;
            endTime = -1;
        } else {
            startTime = -1;
            endTime = time;
        }
        return new EventLog(demandeId, listenerClass, startTime, endTime);
    }
    return null;
}

, .

Answer 4

( ), . , log4j XMLLayout - . , .

. appender, .

, XMLLayout , Apaw benzaw

Answer 5

Log4j LogFilePatternReceiver ...

: 17-11-2011 14:07:14 MyXmlParser -

logformat ( , origin - , "logger" ), , Java SimpleDateFormat dd-MM-yyyy kk: mm: ss

-

... (E ERROR), , .

, ​​ Chainsaw:

http://people.apache.org/~sdeboy

Answer 6

( Java), . log4j.

python script, ( SiteScope - ), .

, , , . , , ;)

Answer 7

Maybe you can write Log4j CustomAppender? For example, as described here: http://mytechattempts.wordpress.com/2011/05/10/log4j-custom-memory-appender/

A user user can use the database or simple Java objects requested by JMX to retrieve statistics. It all depends on how much data you need to save.