C # ServiceStack.Text parsing json stream

I am creating a json deserializer. I am deserializing a rather large json file (25mb) that contains a lot of information. This is an array for words with lots of duplicates. With help, NewtonSoft.JsonI can deserialize the input as a stream:

using (var fs = new FileStream(@"myfile.json", FileMode.Open, FileAccess.Read))
using (var sr = new StreamReader(fs))
using (var reader = new JsonTextReader(sr))
{
    while (reader.Read())
    {
        //Read untill I find the narrow subset I need and start parsing and analyzing them directly
        var obj = JObject.Load(reader); //Analyze this object
    }
}

This allows me to read small parts of json and parse it, check for duplicates, etc.

If I want to do the same with ServiceStack.Text. I am doing something like:

using (var fs = new FileStream(@"myfile.json", FileMode.Open, FileAccess.Read))
using (var sr = new StreamReader(fs))
{
    var result = ServiceStack.Text.JsonSerializer.DeserializeFromReader<MyObject>(sr);
}

MyObject only contains the subset of json that interests me, but this creates significant overhead since I will get a large array containing many duplicates.

In the first method, I can filter them out immediately and therefore not store them in memory.

A trace of memory between the two (this includes the console utilities):

  • NewtonSoft: 30mb
  • ServiceStack.Text: 215mb

:

  • NewtonSoft: 2.5s
  • ServiceStack.Text: 1.5s

, .

, ServiceStack TypeSafety, .

, ServiceStack.Text , , NewtonSoft, ServiceStack.Text?

( , ):

public class MyObject
{
    public List<List<Word>> Words { get; set; }
}

public class Word
{
    public string B { get; set; }
    public string W { get; set; }
    public string E { get; set; }
    public string P { get; set; }
}

( ) 29000 , 8500 . , . , .

+4

Source: https://habr.com/ru/post/1666383/


All Articles