Memory usage Protobuf.net

Heyup. Long time lover protobuf.net.

Quick question. I have a multithreaded C # application that deserializes perhaps 100 objects per second, which is about 50 MB / s. I see very large memory usage, far beyond what I am deserializing. I run the application through the "Red Gate ANTS Memory Profiler" and it shows me a huge number of generation 2 memory objects due to protobuf (over 50% of the applicationโ€™s usage). Most objects are int values โ€‹โ€‹and are associated with:

- TypeModel.TryDeserializeList() - ProtoBuf.Meta.BasicList 

Any help reducing gen 2 memory usage would be appreciated.

Mark

+6
source share
2 answers

It seems to me that the root of T here is the array itself, i.e.

 int[] values = Serializer.Deserialize<int[]>(source); 

If so, it is currently using a slightly suboptimal path for this scenario (for the reason that: using the same code path even on platforms with weak metaprogramming / reflective models such as iOS). I will try to spend several hours getting it at some point, but in answer to your question - you can avoid the problem here by simply adding the parent object:

 [ProtoContract] public class MyDataWrapper { // need a new name... [ProtoMember(1)] public int[] Values { get;set; } } 

and then:

 int[] values = Serializer.Deserialize<MyDataWrapper>(source).Values; 

This is actually fully compatible with data already serialized using Serialize<int[]> if field number 1 . Another advantage of this approach is that you can use the "packed" sub-format if desired (available only for lists / arrays of primitives such as int); although perhaps this is not a great idea in this case due to the large length (this may require buffering during serialization).


Additional context; "v1" here mainly uses MakeGenericType to switch to something like higher on the fly; however, since this approach is not available on many additional platforms for which "v2" is aimed, a less elegant approach is used here. But now that it is fairly stable, I could re-add the optimized version while working on full .NET 2.0 or higher.

+5
source

To clarify Marcs answer I did a quick test

  • Serialization / deserialization using a shell and using an array.
  • With / without GC server

In the test, 100,000 complex objects were created (1 time, 2 doubles, 2 intervals, 2 intervals, a list of lines with 0 and 4 short elements (1 character)) and repeated the serialization / deserialization process 30 times and measured the total time and number of GC collections that occurred during the run. The results were (works in the release outside VS)

 GC IsServer: False, GC latency: Interactive, GC LOH compaction: Default Wrapper serialization Generation 0: 0 collects Generation 1: 0 collects Generation 2: 0 collects Time: 20.363 s ------------------------ Array serialization Generation 0: 0 collects Generation 1: 0 collects Generation 2: 0 collects Time: 30.433 s ------------------------ Wrapper deserialization Generation 0: 109 collects Generation 1: 47 collects Generation 2: 16 collects Time: 71.277 s ------------------------ Array deserialization Generation 0: 129 collects Generation 1: 57 collects Generation 2: 19 collects Time: 89.145 s GC IsServer: True, GC latency: Interactive, GC LOH compaction: Default Wrapper serialization Generation 0: 0 collects Generation 1: 0 collects Generation 2: 0 collects Time: 20.430 s ------------------------ Array serialization Generation 0: 0 collects Generation 1: 0 collects Generation 2: 0 collects Time: 30.364 s ------------------------ Wrapper deserialization Generation 0: 4 collects Generation 1: 3 collects Generation 2: 2 collects Time: 39.452 s ------------------------ Array deserialization Generation 0: 3 collects Generation 1: 3 collects Generation 2: 3 collects Time: 47.546 s 

So my conclusion

  • The approach to the shell benefits both serialization and deserialization (while the latter has a more pronounced effect).
  • The collection of GC data superimposed by the array method is more noticeable when working without a GC server. Also note that the impact of GC performance is very bad when the GC server is not running and deserialized in multiple threads (results not included).

Hope someone finds this helpful.

(unfortunately, the reference code is dependent on the internal code, so I cannot post the full code here).

+1
source

Source: https://habr.com/ru/post/903466/


All Articles