As Mark says, if you use WriteObject overload, which accepts Stream , then bytes written to the stream using the DataContractSerializer will contain the encoded text of the UTF-8 XML document, regardless of the type of Stream instance. If the instance is a MemoryStream , you get an array of bytes in memory (because that's what a MemoryStream is), but these bytes contain UTF-8 encoded XML text - which is usually not described as a binary representation.
If you use WriteObject overload, which accepts XmlWriter , then what you end up with is completely determined by the type of your XmlWriter and how it was initialized. As for the DataContractSerializer , it will call the XmlWriter methods in the appropriate template to describe the XML Infoset representing the state of your object. How this Infoset is encoded depends on the implementation of XmlWriter .
If you had specialized requirements and were especially masochistic, you could implement your own XmlWriter to do any encodings, compress, encrypt, whatever your imagination attracts. Or you can pass an XmlTextWriter that will give you Infoset text encoding (e.g. Stream overload), but with additional options to control character encoding and text formatting. Typically, you pass in an XmlDictionaryWriter , and then you have different encoding options depending on how you create it.
These options include WCF's own binary encoding for XML Infosets: to create an instance with XmlDictionaryWriter.CreateBinaryWriter . Since the WCF team put a lot of effort into developing an effective binary representation to meet WCF performance goals, it seems to me that it will be difficult for you to propose the best option, provided that you can be sure that the result Serialized objects will only need to deserialize .NET code . If you need a binary representation based on a standard standard, you can instead consider the MTOM flavor of XmlDictionaryWriter .