Incremental Buffer Protocol Entry

Question

I have a Buffer protocol for logging data.

message Message { required double val1 = 1; optional int val2 = 2; } message BigObject { repeated Message message = 1; } 

I receive messages once per second. They are stored in memory using my BigObject and are used for some tasks. But at the same time, I want to save these messages in a file for backup in the event of an application failure. A simple BigObject letter will be a waste of time each time. And I'm trying to find a way to write only added messages since the last write to the file. Is there any way to do this?

Answer 1

Protobuf is an advanced format, and your layout is ideal for this. Just open your file at the end and start with a new (empty) BigObject . Add / serialize only a new instance of Message and write to the file (starting from the end).

Now, if you analyze your file from the very beginning, you will get a single BigObject with all Message instances (old and new).

You could do this by registering each individual Message as it arrives, if each time you wrap it in BigObject , that is, in pseudocode

 loop { msg = await NextMessage(); wrapper = new BigObject(); wrapper.Messages.Add(msg); file = OpenFileAtEnd(); wrapper.WriteTo(file); file.Close(); }