Asynchronous serialization of protobuff

Question

The limitation of the protobuf-net implementation is that it invokes the underlying threads synchronously. Not offering an asynchronous API, for example. BeginSerialize / EndSerialize or the equivalent of TPL, we are forced to bind a thread waiting for synchronous I / O of the thread.

Is there any plan to offer asynchronous methods in protobuf-net or, alternatively, any creative ways to solve this problem?

Answer 1

No, this is not currently supported and will be a lot of work.

My suggestion would be: to buffer the data yourself using asynchronous APIs, and then when you have the data, use something like a MemoryStream to deserialize ...

In my defense, I don't know of any other serializer that offers an asynchronous API here. In particular, when it comes to slow / asynchronous streams, this usually means "network": and you usually have a "frame" problem to consider there; protobuf-net will not know your crop requirements ...

Answer 2

You can wait for Task.Run to run synchronous code in the thread pool. This is not the most effective solution, but it is better than blocking. You can even send your own CancellationToken to Task.Run :

 await Task.Run(() => Serializer.SerializeWithLengthPrefix( stream, data, PrefixStyle.Base128), cancellationToken); 

Alternatively, you can use the fairly simple helper method extracted from my JuiceStream library , which I presented as part of the async function request for protobuf-net :

 await ProtobufEx.SerializeWithLengthPrefixAsync( stream, data, PrefixStyle.Base128, cancellationToken); await ProtobufEx.DeserializeWithLengthPrefixAsync<MyType>( stream, PrefixStyle.Base128, cancellationToken); 

Answer 3

I am using protobuff over the network. Although the following solution does not guarantee that it will not block, it makes life much better:

 byte[] emptyByteArray = new Byte[0]; await stream.ReadAsync(emptyByteArray, 0, 0); TaskData d = Serializer.DeserializeWithLengthPrefix<TaskData>(stream, PrefixStyle.Base128); 

Since we check the actual data in the stream, before we start deserializing, it will only be blocked when the stream contains a partial message.

Edit: And we can use a similar trick to serialize:

 MemoryStream mstm = new MemoryStream(); Serializer.SerializeWithLengthPrefix(mstm, data, PrefixStyle.Base128); await stream.WriteAsync(mstm.GetBuffer(), 0, (int)mstm.Position); 

As a bonus, this ensures that it is never blocked.