Bash / unix toolchain binary stream processing / slicing

Question

I have a binary stream at standard input, it is in a fixed size format, a continuous stream of packets, each packet has a header with a length of X and a body with a length of Y.

So, if X = 2 Y = 6, then this is something like 00abcdef01ghijkl02mnopqr03stuvwx , but it is binary, and both headers and data can contain any "characters" (including "\ 0" and a new line), this example is read only .

I want to get rid of the header data so that the result looks like this: abcdefghijklmnopqrstuvwx .

Are there any commands in the unix toolchain that allow me to do this? And are there any tools for processing binary data? The only tool I could think of is od / hexdump , but how do you convert the result back to binary?

Answer 1

Use xxd , which goes to and from hexdump.

 xxd -c 123 -ps 

will output your stream with 123 bytes per line. For reverse use

 xxd -r -p 

Now you can match this with cut to discard characters as you can do something like

 cut -c 3- 

to get all characters from 3 to the end of the string. Remember to use multiple 2X characters to account for two hexadecimal characters in bytes.

So something like strings

 xxd -c X+Y -ps | cut -c 2X+1- | xxd -r -p 

where X+Y and 2X+1 replaced by actual numerical values. You will need to put your data stream in the appropriate place in the above command.

Answer 2

Perl is a fairly standard unix tool. Pipe it to perl. If its fixed-length byte is aligned, the simple substr operation will work. Here is an example of perl that should work.

 #!/usr/bin/env perl use strict; use warnings; my $buf; my $len = 8; my $off = 2; while(sysread(STDIN,$buf,$len) != 0 ){ print substr($buf,$off); } exit 0; 

Answer 3

As a single line, I would write:

 perl -00 -ne 'chomp; while (/(?:..)(......)/sg) {print $1}' 

Example:

 echo '00abcdef01ghijkl02mnopqr03stuvw 00abcdef01ghi kl02mnopqr' | perl -00 -ne 'chomp; while (/(?:..)(......)/sg) {print $1}' | od -c 

produces

 0000000 abcdefghijklmnop 0000020 qrstuvw \nabcdefgh 0000040 i \nklmnopqr 0000052 

Answer 4

Here also bbe - binary block editor , which is a binary sed for processing binary data in a Unix way.

http://bbe-.sourceforge.net

Answer 5

The binary stream editor is a tool written in java to process streams. It can be used both from java and from the command line. https://sourceforge.net/projects/bistreameditor/

DISCLAIMER: I am the author of this tool.

Unlike newline tools, such as sed, it allows selective movement and storage of data through traversal and buffer. Binary data can be treated as single byte characters and allowed string / match operations. It can record on multiple outputs and use different encodings. Because of this flexibility, there are currently many options on the command line that need to be simplified.

The bse.zip file should be downloaded and used. For the above example, we just need to do substr (2) at the input of len 8. Full command line

 java -classpath "./bin:$CMN_LIB_PATH/commons-logging-1.1.1.jar:$CMN_LIB_PATH/commons-io-2.1.jar:$CMN_LIB_PATH/commons-jexl-2.1.1.jar:$CMN_LIB_PATH/commons-lang3-3.1.jar" -Dinputsrc=file:/fullpathtofile|URL|System.in -Dtraverser=org.milunsagle.io.streameditor.FixedLengthTraverser -Dtraversercons=size -Dtraverserconsarg0=8 -Dbuffer=org.milunsagle.io.streameditor.CircularBuffer -Dbuffercons=size -Dbufferconsarg0=8 -Dcommands='PRN V $$__INPUT.substring(2)' org.milunsagle.io.streameditor.BinaryStreamEditorInvoker