How to find broken NMEA log sentences using grep?

Question

My GPS logger periodically leaves "incomplete" lines at the end of the log files. I think they are only at the end, but I want to check all the lines just in case.

The full sentence of the offer looks like this:

$GPRMC,005727.000,A,3751.9418,S,14502.2569,E,0.00,339.17,210808,,,A*76 

The string must begin with the $ character and end with the * character and a two-character hexadecimal checksum. I do not care about the correctness of the checksum, just that it is present. It is also necessary to ignore the "ADVER" clauses, which do not have a checksum and are located at the beginning of each file.

The following Python code may work:

 import re from path import path nmea = re.compile("^\$.+\*[0-9A-F]{2}$") for log in path("gpslogs").files("*.log"): for line in log.lines(): if not nmea.match(line) and not "ADVER" in line: print "%s\n\t%s\n" % (log, line) 

Is there a way to do this with grep or awk or something simple? I really didn't understand how to get grep to do what I want.

Update : thanks @Motti and @Paul, I was able to do the following to do almost what I wanted, but had to use single quotes and remove the trailing $ before it would work

 grep -nvE '^\$.*\*[0-9A-F]{2}' *.log | grep -v ADVER | grep -v ADPMB 

Two more questions arise: how can I ignore empty lines? And can I combine the last two grep s?

Answer 1

Minimal testing shows that this should be done:

 grep -Ev "^\$.*\*[0-9A-Fa-f]{2}$" a.txt | grep -v ADVER 

• -E use extended regular expression
• -v Show lines that do not match
• ^ starts with
• . * anything
• \ * asterisk
• [0-9A-Fa-f] hexadecimal digit
• {2} exactly two of the previous
• $ end of line
• | grep -v ADVER cut off ADVER lines

Hth, motty.

Answer 2

@Motti's answer does not ignore the ADVER lines, but you can easily link the results of this grep to another:

 grep -Ev "^\$.*\*[0-9A-Fa-f]{2}$" a.txt |grep -v ADVER 

Answer 3

@Tom (paraphrased) I had to remove the trailing $ for it to work

Removing $ means that the line may end with something else (for example, the following will be accepted)

 $GPRMC,005727.000,A,3751.9418,S,14502.2569,E,0.00,339.17,210808,,,A*76xxx 

@Tom And can I combine the last two greps?

 grep -Ev "ADVER|ADPMB" 

Answer 4

@Motti: grep union does not work, it has no effect.

I understand that without the final $ , something else could be in the checksum and still match, but it didn’t work out with it, so I had no choice ...

GNU grep 2.5.3 and GNU bash 3.2.39 (1), if that matters.

And it looks like the log files are using DOS breaks (CR + LF). Does grep need a switch for proper handling?

Answer 5

@tom

GNU grep 2.5.3 and GNU bash 3.2.39 (1), if that matters. And it looks like the DOS breaks (CR + LF) are used in the log files. Does grep require a key to handle it properly?

I use grep (GNU grep) 2.4.2 on Windows (for shame!) And it works for me (and DOS linear interrupts are naturally accepted), at the moment I don't have access to other OSs, so I'm sorry, but I'm not I can help you: o (