Filter log file by defining regular expressions

Question

I have some HUGE log files (50 MB, ~ 500 thousand lines). I need to start filtering some of the crap. Log files are created using log4j and have a basic pattern:

[log-level] date-time class etc, etc log-message 

I am looking for a way in which I can determine the beginning and beginning of a regular expression (or something similar) that will filter out the relevant entries from a file so that I can more easily get through these massive files. My thoughts are that the initial regular expression will be the log level, and the final regular expression will be something in the log message. I am sure that I can write a java program to accomplish this task, but I thought I would ask the community before taking this path. Thanks in advance.

---

Let me tell you about my question. Suppose there is the following snippet in my log file:

 [DEBUG] date-time class etc, etc log-message-1 [WARN] date-time class etc, etc log-message-2 [DEBUG] date-time class etc, etc log-message-3 [DEBUG] date-time class etc, etc log-message-1 [WARN] date-time class etc, etc log-message-2 [DEBUG] date-time class etc, etc log-message-6 

I need a way to filter logEntry1 and logEntry2, so I get:

 [DEBUG] date-time class etc, etc log-message-3 [DEBUG] date-time class etc, etc log-message-6 

I would hope that this would be the definition of some sets of regular expression pattern pairs. In my example above, I would like to define a pair for logEntry1 and another for logEntry2.

Hope this helps clarify my question.

Answer 1

Assuming log-message-1 and log-message-2 and unique patterns.

 $ awk -vRS= '!/log-message-[12]/' ORS="\n\n" file [DEBUG] date-time class etc, etc log-message-3 [DEBUG] date-time class etc, etc log-message-6 

Answer 2

 (zyx:~) % echo $T [DEBUG] date-time class etc, etc log-message-1 [WARN] date-time class etc, etc log-message-2 [DEBUG] date-time class etc, etc log-message-3 [DEBUG] date-time class etc, etc log-message-1 [WARN] date-time class etc, etc log-message-2 [DEBUG] date-time class etc, etc log-message-6 (zyx:~) % echo $T | perl -e '$_=join("", <>); s/\[DEBUG\][^\n]*\n(log-message-1|log-message-2).*?(?=\n\[(DEBUG|WARN)\]|$)//sg; s/\[WARN\].*?(?=\n\[(DEBUG|WARN)\]|$)//sg; print;' [DEBUG] date-time class etc, etc log-message-3 [DEBUG] date-time class etc, etc log-message-6 

Answer 3

Use awk or awk-style perl with one layer.