Can I unzip and deserialize a file using streams?

Question

My application serializes the object using Json.Net, compresses the received JSON, and then saves it to a file. In addition, an application can load an object from one of these files. These objects can be tens of MB in size, and I am worried about memory usage due to the way the existing code creates large strings and byte arrays: -

public void Save(MyClass myObject, string filename) { var json = JsonConvert.SerializeObject(myObject); var bytes = Compress(json); File.WriteAllBytes(filename, bytes); } public MyClass Load(string filename) { var bytes = File.ReadAllBytes(filename); var json = Decompress(bytes); var myObject = JsonConvert.DeserializeObject<MyClass>(json); } private static byte[] Compress(string s) { var bytes = Encoding.Unicode.GetBytes(s); using (var ms = new MemoryStream()) { using (var gs = new GZipStream(ms, CompressionMode.Compress)) { gs.Write(bytes, 0, bytes.Length); gs.Close(); return ms.ToArray(); } } } private static string Decompress(byte[] bytes) { using (var msi = new MemoryStream(bytes)) { using (var mso = new MemoryStream()) { using (var gs = new GZipStream(msi, CompressionMode.Decompress)) { gs.CopyTo(mso); return Encoding.Unicode.GetString(mso.ToArray()); } } } } 

I was wondering if Save / Load methods can be replaced by threads? I found examples of using threads with Json.Net, but I'm struggling to figure out how to fit into the extra compression material.

Answer 1

JsonSerializer has serialization methods from JsonTextReader and StreamWriter that can be created on top of any stream, including GZipStream . Using them, you can create the following extension methods:

 public static class JsonExtensions { // Buffer sized as recommended by Bradley Grainger, https://faithlife.codes/blog/2012/06/always-wrap-gzipstream-with-bufferedstream/ const int BufferSize = 8192; public static void SerializeToFileCompressed(object value, string path, JsonSerializerSettings settings) { using (var fs = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.Read)) SerializeCompressed(value, fs, settings); } public static void SerializeCompressed(object value, Stream stream, JsonSerializerSettings settings) { using (var compressor = new GZipStream(stream, CompressionMode.Compress)) using (var writer = new StreamWriter(compressor, Encoding.UTF8, BufferSize)) { var serializer = JsonSerializer.CreateDefault(settings); serializer.Serialize(writer, value); } } public static T DeserializeFromFileCompressed<T>(string path, JsonSerializerSettings settings = null) { using (var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read)) return DeserializeCompressed<T>(fs, settings); } public static T DeserializeCompressed<T>(Stream stream, JsonSerializerSettings settings = null) { using (var compressor = new GZipStream(stream, CompressionMode.Decompress)) using (var reader = new StreamReader(compressor)) using (var jsonReader = new JsonTextReader(reader)) { var serializer = JsonSerializer.CreateDefault(settings); return serializer.Deserialize<T>(jsonReader); } } } 

See Performance Tips: Optimize Memory Usage in the Json.NET Documentation.

Answer 2

For those who are looking for the idea of ​​using extensions from @dbc in uwp applications, I changed the code to this - where StorageFile is the file you have write access to.

 public static async void SerializeToFileCompressedAsync(object value, StorageFile file, JsonSerializerSettings settings = null) { using (var stream = await file.OpenStreamForWriteAsync()) SerializeCompressed(value, stream, settings); } public static void SerializeCompressed(object value, Stream stream, JsonSerializerSettings settings = null) { using (var compressor = new GZipStream(stream, CompressionMode.Compress)) using (var writer = new StreamWriter(compressor)) { var serializer = JsonSerializer.CreateDefault(settings); serializer.Serialize(writer, value); } } public static async Task<T> DeserializeFromFileCompressedAsync<T>(StorageFile file, JsonSerializerSettings settings = null) { using (var stream = await file.OpenStreamForReadAsync()) return DeserializeCompressed<T>(stream, settings); } public static T DeserializeCompressed<T>(Stream stream, JsonSerializerSettings settings = null) { using (var compressor = new GZipStream(stream, CompressionMode.Decompress)) using (var reader = new StreamReader(compressor)) using (var jsonReader = new JsonTextReader(reader)) { var serializer = JsonSerializer.CreateDefault(settings); return serializer.Deserialize<T>(jsonReader); } }