Log analysis using sed or grep

Question

I want to capture data from this type of log.

November 12, 13:46:14 Home cxxd [8892]: 208 11/12 13: 46: 14 | qc = IN (1), qt = A (1), query = "www.yahoo.com."

Implemented this, which gives me a URL. But it doesn’t work with "TAIL -F" so that I can only control live URLs.

tail -100 / var / log / system.log | grep "query =" | sed -e "s /.* query = //" | sed -e "s / \" // g "| sed -e" s /.$// "

Please suggest or improve

Answer 1

I expect your multiple sed scripts to work with tail -F output, just not the way you expect.

C standard I / O libraries will perform buffering to improve performance. The IO library can perform (a) no buffering (b) row buffering (c) block buffering. A buffer line is usually selected if the output goes to the terminal. But if the output goes to a file or a channel, then block buffering is usually chosen. (This is more complicated than this — the behavior changes if the file descriptor in question is used for stdout or stderr or another file. See setvbuf(3) more details.)

So, although the block buffering that you see now is probably better for performance, it means you can wait a while before ever seeing any output, as each command ultimately accumulates a data block. At the very least, grep(1) allows you to use the --line-buffered command line --line-buffered to use line buffering - and sed(1) allows the --unbuffered command line --unbuffered output buffers more often. So try the following:

 tail -f /var/log/system.log | grep --line-buffered "query=" | sed -u -e "s/.*query=//" | sed -u -e "s/\"//g" | sed -u -e "s/.$/ /" 

(I did not find similar parameters for tail(1) , but even if it sends data blocks to others, changes in grep(1) and sed(1) will help a lot.)

Answer 2

Try reducing the number of channels by replacing multiple calls with grep and sed with awk :

 tail -f /var/log/system.log | awk -F'=' '/query=/ { sub(/^"/, "", $NF); sub(/."$/, "", $NF); print $NF }' 

... which takes each line corresponding to "query =" and captures everything after the last "=", replaces the first "and the final". "and prints the result.

Answer 3

Try tail -f and grep -line-buffered