How to search lines in a file between two timestamps using Bash

Question

In bash, I am trying to read a log file and print only lines with a timestamp between two specific times. Time format: hh: mm: ss. For example, I would look for lines that will fall between 12:52:33 to 12:59:33.

I want to use regex because I can use it in grep function.

Each log in begins with some_nr 2014-05-15 21:58:00,000000 rest_of_line .

My solution gives me lines with an edge of 1 min. I cut out ss and took all the lines with hh:mm:[0-9]{2} . $ 2 has the format filename_hh:mm:; for example: "24249_16:05:;24249_16:05:;24249_16:07:;24249_16:07:;24249_16:08:"

My code is:

 B=$2 for line in ${B//;/ } ; do TENT=`echo $line | awk '{split($0,numbers,"_"); print numbers[1]}'`"_logs.txt" TIME=`echo $line | awk '{split($0,numbers,"_"); print numbers[2]}'`"[0-9]{2}" grep -iE ${TIME} ${TENT} >> ${FILE1} done 

I need a solution with an edge of 15 seconds at any time, not 60. I want to have input in the format filename_hh:mm:ss and take the lines for hh: mm: ss +/- 15s or filename_hh:mm:ss(1)_hh:mm:ss(2) and take the lines between hh: mm: ss (1) and hh: mm: ss (2). For some time there are no lines, so the solution should “recognize” if it sometimes coincides with the entered interval or not.

The log files are as follows:

 1002143 1002143 2014/15/05 22:09:52.937004 bla 1002130 2014/15/05 22:09:44.786002 bla bla 1001667 2014/15/05 22:09:44.592009 bl a bla 1001667 1001667 2014/15/05 22:09:44.592009 bl a bla 

Answer 1

I believe sed is the best option:

 sed -rne '/<timestamp>/,/<timestamp>/ p' <file> 

Example:

  tiago@dell : ~ $ sed -rne '/ 08: 17: 38 /, / 08: 24: 36 / p' / var / log / syslog 
 May 16 08:17:38 dell AptDaemon.Worker: INFO: Processing transaction / org / debian / apt / transaction / 08a244f7b8ce4fad9f6b304aca9eae7a
 May 16 08:17:50 dell AptDaemon.Worker: INFO: Finished transaction / org / debian / apt / transaction / 08a244f7b8ce4fad9f6b304aca9eae7a
 May 16 08:18:50 dell AptDaemon.PackageKit: INFO: Initializing PackageKit transaction
 May 16 08:18:50 dell AptDaemon.Worker: INFO: Simulating trans: / org / debian / apt / transaction / 37c3ef54a6ba4933a561c49b3fac5f6e
 May 16 08:18:50 dell AptDaemon.Worker: INFO: Processing transaction / org / debian / apt / transaction / 37c3ef54a6ba4933a561c49b3fac5f6e
 May 16 08:18:51 dell AptDaemon.PackageKit: INFO: Get updates ()
 May 16 08:18:52 dell AptDaemon.Worker: INFO: Finished transaction / org / debian / apt / transaction / 37c3ef54a6ba4933a561c49b3fac5f6e
 May 16 08:24:36 dell AptDaemon: INFO: Quitting due to inactivity

Answer 2

The log file is usually sorted by timestamp, suppose the timestamp is in the first column, you can:

 awk -v from="12:52:33" -v to="12:59:33" '$1>=from && $1<=to' foo.log 

this way you can change from and to to get a different set of log entries. regex is not a good tool for calculating and comparing numbers.

Answer 3

You can use this regular expression in egrep :

 egrep '12:5[2-9]:33' file.log 

Answer 4

The wrong tool is used for this task. If you have a regular expression like the one given by @anubhava, you can easily find a time interval that doesn't match it. grep and regular expressions may work in a few special cases, but they do not scale in the general case.

Can you use some kind of tool that can "understand" timestamps? Probably every scripting language (perl, python, ruby, lua) has built-in or library support for the time and date of parsing.

However, you can use the powers of GNU date:

 % date --date="2014-05-15 21:58:00 15 sec ago" +'%Y-%m-%d %H:%M:%S' 2014-05-15 21:57:45 % date --date="2014-05-15 21:58:00 15 sec" +'%Y-%m-%d %H:%M:%S' 2014-05-15 21:58:15 

and connect this to the idea of ​​a Tiago sed filter.

Answer 5

You can try the following perl script:

 #! /usr/bin/perl use warnings; use strict; use Time::Piece; use autodie; my $arg=shift; my @a =split("_",$arg); my $fn=shift @a; my $dfmt='%Y/%d/%m'; my $fmt=$dfmt.' %H:%M:%S'; my $t = localtime; my $date=$t->strftime($dfmt); my $t1; my $t2; if (@a == 1) { my $d=$date.' '.$a[0]; my $tt=Time::Piece->strptime($d, $fmt); $t1=$tt-15; $t2=$tt+15; } elsif (@a == 2) { $t1=Time::Piece->strptime($date.' '.$a[0], $fmt); $t2=Time::Piece->strptime($date.' '.$a[1], $fmt); } else { die "Unexpected input argument!"; } $fn=$fn.'_logs.txt'; doGrep($fn,$t1,$t2,$fmt); sub doGrep { my ($fn,$t1,$t2,$fmt) = @_; open (my $fh, "<", $fn); while (my $line=<$fh>) { my ($d1,$d2) = $line=~/\S+\s+(\S+)\s+(\d\d:\d\d:\d\d)/; my $d=$d1.' '.$d2; my $t=Time::Piece->strptime($d, $fmt); print $line if ($t>$t1 && $t<$t2); } close ($fh); } 

Run it from the command line using the syntax: ./p.pl A_22:09:14 .