Binary format for transmitting tabular data

Question

Binary format for transmitting tabular data

I support an outdated embedded device that interacts with the real world. Generally speaking, this device collects data from sensors, processes data using an internal algorithm, and displays a warning when the data reaches a certain “bad” state.

For debugging purposes, we would like this device to regularly send us many of the received data, as well as data after its processing.

We concluded that most of the data can be described in tabular form, something like rows

sensor|time|temprature|moisture
------+----+----------+--------
1     |3012|20        |0.5
2     |3024|22        |0.9

We obviously need to support more than one form of table.

So, basically, we need a protocol that can take a certain set of table descriptions, and then provide the table data in accordance with its description.

Example pseudo-code for sending data:

table_t table = select_table(SENSORS_TABLE);
sensors_table_data_t data[] = {
    {1,3012,20,0.5},
    {1,3024,22,0.9}
    };
send_data(table,data);

An example of pseudo code for receiving data:

data_t *data = recieve();
switch (data->table) {
    case SENSORS_TABLE:
         puts("sensor|time|temprature|moisture");
         for (int i=0;i<data->length;i++) printf(
             "%5s|%4s|%9s|%9s\n",
              data->cell[i]->sensor,
              data->cell[i]->time,
              data->cell[i]->temprature,
              data->cell[i]->moisture);
         break;
    case USER_INPUT_TABLE:
         ...
}

The definition of tables can be performed offline both on the device and on the client computer that is exchanging with it or on the Internet. We can add a simple communication protocol to harmonize the table format at boot time.

Since it is an obsolete device, it only supports RS232 communication, and since its processor is rather slow (equivalent to 486), we cannot afford to use any data transfer methods like XML. It is too expensive (either in time or in width). Sending raw SQL commands was also considered and rejected due to bandwidth considerations.

[edit]

, , . , , .

, , , , .

, Google, , C.

[/]

, , ? ?

, , :

1) : , . .

2) : ( ), . .

. , .

+3

c sql database embedded protocols

Elazar Leibovich 19 . '09 5:48

8

, .

http://code.google.com/p/protobuf/

- . Google RPC .

, protobufs , . , / . API (, protobuf python).

+2

ramanujan 19 . '09 8:25

, . , . , :

sensor|time|temprature|moisture
------+----+----------+--------
1     |3012|20        |0.5

:

0x01 0x0B 0xC4 0x14 [4 bytes for float 0.5]

, 4 (float) . .

, . .

, , . , :

[STX] message [ETX]

ASCII STX ETX (, 0x02 0x03). , . . 0x02 0x03, . 0x02 . 0x02 0x03 .

, , .

, PPP.

+2

kgiannakakis 21 . '09 8:47

, . , , , ( ). , , . "" , , , .

+1

gbarry 19 . '09 7:31

CSV (. RFC 4180 CSV), (. gbarry).

RFC ( 2, 3), .

, CSV-, "" .

+1

bortzmeyer 19 . '09 9:23

-:

[header][data][checksum]

, :

[header][table_id][elements][data][checksum]

[header]   : start of frame
[table_id] : table
[elements] : payload size
[data]     : raw data
[checksum] : checksum/crc, just to be on the safe side

"" .

, .

EDIT:

- -, /. ?

, . 4 10 40%, 1,6% 256 . , .

+1

Marcelo MD 22 . '09 17:10

, , , B64. . - 1/3. . . . . :

/********************************************************************/
/*                                                                  */
/* Functions:                                                       */
/* ----------                                                       */
/* TBase64Encode()                                                  */
/* TBase64Decode()                                                  */
/* TBase64EncodeBlock()                                             */
/* TBase64DecodeBlock()                                             */
/*                                                                  */
/********************************************************************/

#include "yourstuff.h"


// This table is used to encode 6 bit binary to Base64 ASCII.
static char Base64Map[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdef"
                          "ghijklmnopqrstuvwxyz0123456789+/";

// This table is used to decode Base64 ASCII back to 6 bit binary.
static char Base64Decode[]=
{
    62,                                         // '+'
    99, 99, 99,                                 // **** UNUSED ****
    63,                                         // '/'
    52, 53, 54, 55, 56, 57, 58, 59, 60, 61,     // '0123456789'
    99, 99, 99, 99, 99, 99, 99,                 // **** UNUSED ****
    0, 1, 2, 3, 4, 5, 6, 7, 8, 9,               // 'ABCDEFGHIJ'
    10, 11, 12, 13, 14, 15, 16, 17, 18, 19,     // 'KLMNOPQRST'
    20, 21, 22, 23, 24, 25,                     // 'UVWXYZ'
    99, 99, 99, 99, 99, 99,                     // **** UNUSED ****
    26, 27, 28, 29, 30, 31, 32, 33, 34, 35,     // 'abcdefghij'
    36, 37, 38, 39, 40, 41, 42, 43, 44, 45,     // 'klmnopqrst'
    46, 47, 48, 49, 50, 51                      // 'uvwxyz'
};




/** Convert binary data to Base64 data.
 *
 * @return  Size of output buffer if ok, -1 if problem (invalid paramaters).
 *
 * @param   input  - Pointer to input data.
 * @param   size   - Number of bytes to encode.
 * @param   output - Pointer to output buffer.
 *
 * @note    Up to caller to ensure output buffer is big enough. As a rough
 *          guide your output buffer should be (((size/3)+1)*4) bytes.
 */
int TBase64Encode( const BYTE *input, int size, PSTR output)
{
    int i, rc=0, block_size;

    while (size>0)
    {
        if (size>=3)
            block_size = 3;
        else
            block_size = size;

        i = TBase64EncodeBlock( input, block_size, output);

        if (i==-1)
            return -1;

        input += 3;
        output += 4;
        rc += 4;
        size -= 3;
    }

    return rc;
}




/** Convert Base64 data to binary data.
 *
 * @return  Number of bytes in output buffer, negative number if problem
 *          as follows:
 *           -1 : Invalid paramaters (bad pointers or bad size).
 *           -2 : Outside of range value for Base64.
 *           -3 : Invalid base 64 character.
 *
 * @param   input  - Pointer to input buffer.
 * @param   size   - Size of input buffer (in bytes).
 * @param   output - Pointer to output buffer.
 *
 * @note    Up to caller to ensure output buffer is big enough. As a rough
 *          guide your output buffer should be (((size/4)+1)*3) bytes.
 *          NOTE : The input size paramater must be multiple of 4 !!!!
 *          Note that error codes -2 and -3 essentiallty mean the same
 *          thing, just for debugging it means something slight different
 *          to me :-). Calling function can just check for any negative
 *          response.
 */
int TBase64Decode( CPSTR input, int size, BYTE *output)
{
    int output_size=0, i;

    // Validate size paramater only.
    if (size<=0 || size & 3)
        return -1;

    while (size>0)
    {   
        i = TBase64DecodeBlock( input, output);
        if (i<0)
            return i;

        output_size += i;
        output += i;
        input += 4;
        size -= 4;
    }

    return output_size;
}




/** Convert up to 3 bytes of binary data to 4 bytes of Base64 data.
 *
 * @return  0 if ok, -1 if problem (invalid paramaters).
 *
 * @param   input  - Pointer to input data.
 * @param   size   - Number of bytes to encode(1 to 3).
 * @param   output - Pointer to output buffer.
 *
 * @note    Up to caller to ensure output buffer is big enough (4 bytes).
 */
int TBase64EncodeBlock( const BYTE *input, int size, PSTR output)
{
    int i;
    BYTE mask;
    BYTE input_buffer[3];

    // Validate paramaters (rudementary).
    if (!input || !output)
        return -1;
    if (size<1 || size>3)
        return -1;

    memset( input_buffer, 0, 3);
    memcpy( input_buffer, input, size);

    // Convert three 8bit values to four 6bit values.
    mask = input_buffer[2];
    output[3] = mask & 0x3f;            // Fourth byte done...

    output[2] = mask >> 6;
    mask = input_buffer[1] << 2;
    output[2] |= (mask & 0x3f);         // Third byte done...

    output[1] = input_buffer[1] >> 4;
    mask = input_buffer[0] << 4;
    output[1] |= (mask & 0x3f);         // Second byte done...

    output[0] = input_buffer[0]>>2;     // First byte done...

    // TEST
//  printf("[%02x,%02x,%02x,%02x]", output[0], output[1], output[2], output[3]);

    // Convert 6 bit indices to base64 characters.
    for (i=0; i<4; i++)
        output[i] = Base64Map[output[i]];

    // Handle special padding.
    switch (size)
    {
        case 1:
            output[2] = '=';
        case 2:
            output[3] = '=';
        default:
            break;
    }


    return 0;
}




/** Convert 4 bytes of Base64 data to 3 bytes of binary data.
 *
 * @return  Number of bytes (1 to 3) if ok, negative number if problem
 *          as follows:
 *           -1 : Invalid paramaters (bad pointers).
 *           -2 : Outside of range value for Base64.
 *           -3 : Invalid base 64 character.
 *
 * @param   input  - Pointer to input buffer (4 bytes).
 * @param   output - Pointer to output buufer (3 bytes).
 *
 * @comm    While there may be 1, 2 or 3 output bytes the output
 *          buffer must be 3 bytes. Note that error codes -2 and -3
 *          essentiallty mean the same thing, just for debugging it
 *          means something slight different to me :-). Calling function
 *          can just check for any negative response.
 */
int TBase64DecodeBlock( CPSTR input, BYTE *output)
{
    int i, j;
    int size=3;
    BYTE mask;
    BYTE input_buffer[4];

    // Validate paramaters (rudementary).
    if (!input || !output)
        return -1;

    memcpy( input_buffer, input, 4);

    // Calculate size of output data.
    if (input_buffer[3]=='=')
    {
        input_buffer[3] = 43;
        size--;
    }
    if (input_buffer[2]=='=')
    {
        input_buffer[2] = 43;
        size--;
    }

    // Convert Base64 ASCII to 6 bit data.
    for (i=0; i<4; i++)
    {
        j = (int) (input_buffer[i]-43);
        if (j<0 || j>79)
            return -2;          // Invalid char in Base64 data.
        j = Base64Decode[j];
        if (j==99)      
            return -3;          // Invalid char in Base64 data.

        input_buffer[i] = (char) j;
    }

    // TEST
//  printf("[%02x,%02x,%02x,%02x]", input_buffer[0], input_buffer[1], input_buffer[2], input_buffer[3]);

    // Convert four 6bit values to three 8bit values.
    mask = input_buffer[1] >> 4;
    output[0] = (input_buffer[0]<<2) | mask;    // First byte done.

    if (size>1)
    {
        mask = input_buffer[1] << 4;
        output[1] = input_buffer[2] >> 2;
        output[1] |= mask;              // Second byte done.

        if (size==3)
        {
            mask = input_buffer[2] << 6;
            output[2] = input_buffer[3] | mask;     // Third byte done.
        }
    }

    return size;
}

+1

Tim Ring 05 '09 12:19

...

[header] [data] [check-sum]

[data] - , [header] [ ] . [] [ ].

Now reduces overloading [header], [checksum], making a certain useful data chain.

After reading the data, read and show the data in any format by doing something from your PC (which will be your debug PC ..)

0

Alphaneo Apr 20 '09 at 5:09

source share

poundifdef · Accepted Answer · 2009-04-19T06:05:12+0000

, ( , ).

, : ?

:

struct table_format_header {
  int number_of_fields; /* number of fields that will be defined in table */
                        /* sent before the field descriptions themselves  */
};

struct table_format {
   char column_name[8];   /* name of column ("sensor");  */
   char fmt_specifier[5]; /* format specifier for column */

   ... (etc)
}

/ (- ), header, , table_format structs . , - , , . , (int precision:3),

Binary format for transmitting tabular data

More articles: