OutOfMemoryException when serializing a list of objects containing a large array of bytes

Question

I am trying to serialize a fairly large amount of data with protobuf.net. I have problems with OutOfMemoryException . I am trying to pass data using IEnumerable<DTO> so as not to use too much memory. Here is a simplified version of the program that should cause an error:

 class Program { static void Main(string[] args) { using (var f = File.Create("Data.protobuf")) { ProtoBuf.Serializer.Serialize<IEnumerable<DTO>>(f, GenerateData(1000000)); } using (var f = File.OpenRead("Data.protobuf")) { var dtos = ProtoBuf.Serializer.DeserializeItems<DTO>(f, ProtoBuf.PrefixStyle.Base128, 1); Console.WriteLine(dtos.Count()); } Console.Read(); } static IEnumerable<DTO> GenerateData(int count) { for (int i = 0; i < count; i++) { // reduce to 1100 to use much less memory var dto = new DTO { Data = new byte[1101] }; for (int j = 0; j < dto.Data.Length; j++) { // fill with data dto.Data[j] = (byte)(i + j); } yield return dto; } } } [ProtoBuf.ProtoContract] class DTO { [ProtoBuf.ProtoMember(1, DataFormat=ProtoBuf.DataFormat.Group)] public byte[] Data { get; set; } } 

Interestingly, if you reduce the size of the array on each DTO to 1100, the problem will disappear! In my actual code, I would like to do something similar, but this is an array of floats that I will serialize, not bytes. Notabene I think you can skip filling out some of the data to speed up this problem.

This is using protobuf version 2.0.0.594. Any help would be greatly appreciated!

EDIT:

The same problem is observed with version 2.0.0.480. Code will not work with version 1.0.0.280.

Answer 1

to; it was some unfortunate moment - basically, he checked whether it should be cleared whenever the buffer is full, and, as a result, being in the middle of writing an element with a prefix of length, he could not take a picture correctly at this point, I added a setting so that whenever it detects that it reaches an erasable state, and there is something worth washing up (currently 1024 bytes), then it will be activated more aggressively. It was made as r597. With this patch, it now works as expected.

At the same time, there is a way to avoid this failure without changing the version: iterating over the data in the source, serializing each individually using SerializeWithLengthPrefix with the base of the 128 prefix and the field number 1; this is 100% identical in terms of what goes through the wire, but has a separate serialization cycle for each:

 using (var f = File.Create("Data.protobuf")) { foreach(var obj in GenerateData(1000000)) { Serializer.SerializeWithLengthPrefix<DTO>( f, obj, PrefixStyle.Base128, Serializer.ListItemTag); } } 

Thank you for noticing; p

Answer 2

It seems you are overcoming the 1.5 GB limit: Allocating more than 1000 MB of memory in a 32-bit .NET process

You already noticed that when you reduce the sample size, your application works fine. This is not a problem with protobuf (I suppose), but with your attempt to create an array that needs to allocate more than 1.5 GB of memory.

Update

Here is a simple test:

 byte[] data = new byte[2147483648]; 

This should raise an OutOfMemoryException , just like this:

 byte[][] buffer = new byte[1024][]; for (int i = 0; i < 1024; i++) { buffer[i] = new byte[2097152]; } 

Something pushes your data bytes into an adjacent container larger than 1.5 GB.