How to choose a file format?

Question

I created the .NET application many years ago without thinking about the file format: it uses soap formatting to serialize our large hierarchy of objects. It was dirty, and so I did not think about it.

Now I'm trying to create a more optimal file format, given the following problem: When the file is saved, it will eventually be converted to an array of bytes and sent to the database for storage by posting. This becomes a big problem because you have all your objects in memory, then you allocate more memory for the serializer, and then allocate even more memory for the byte array. Even modest object graphs end up using large amounts of memory to take care of saving the file.

I'm not sure how to improve this both in terms of file format and potentially in terms of algorithm (objects → stream → byte array)

UPDATE: I always locked an array of bytes before sending it by cable, so while this good advice, it has already been implemented in my application.

I converted from Soap to Binary Serialization, and that made a huge difference: our files are about 7x smaller than before. (Of course, your mileage may vary).

Answer 1

If you need efficient serialization, and anyway, if it is serialized as a binary format, just use standard binary serialization in .NET. You can simply decorate serializable types with the [Serializable] attribute and use BinaryFormatter to serialize your objects into bytes [].

Answer 2

one very quick solution if you have not tried it. This will not completely reduce overhead, but will help.

When serializing objects, use attributes instead of nodes. Nodes use a lot of free space. You can easily accomplish this by adding the [XmlAttribute] tag above the property / field.

Link Link: http://msdn.microsoft.com/en-us/library/2baksw0z(VS.71).aspx

Answer 3

BinaryFormatter + DeflateStream = Compressed Saved Objects

using System; using System.IO; using System.IO.Compression; using System.Runtime.Serialization.Formatters.Binary; namespace CompressedSerialized { class Program { static void Main(string[] args) { var obj1 = new MyObject() { Prop1 = "p1", Prop2 = "p2" }; MyObject obj2 = null; var bin = new BinaryFormatter(); byte[] buffer = null; using (var ms = new MemoryStream()) { using (var zip = new DeflateStream(ms, CompressionMode.Compress)) { bin.Serialize(zip, obj1); zip.Flush(); } buffer = ms.ToArray(); } using (var ms = new MemoryStream(buffer)) using (var unzip = new DeflateStream(ms, CompressionMode.Decompress)) { var des = bin.Deserialize(unzip); obj2 = des as MyObject; } } } [Serializable] public class MyObject { public string Prop1 { get; set; } public string Prop2 { get; set; } } } 

Answer 4

You can also try using compressed / compressed stream, I think from memory SharpZipLib allows you to create compressed streams.

Answer 5

Why not migrate your application from XML to JSON? There are a number of libraries that can serialize / deserialize JSON in .NET.

Answer 6

I used LZMA to compress the data that I store in the database. For example, get things from 36,000 to 6,000. It is really easy to use, and you do not need to have data stored as binary files, it can also be a string.