Why is my DeflateStream not receiving data correctly over TCP?

Question

I have a TcpClient class on configuring client and server on my local machine. I use network stream to facilitate messaging between the two successfully.

Moving Forward I'm trying to implement compression in messages. I tried GZipStream and DeflateStream. I decided to focus on DeflateStream. However, the connection now hangs without reading data.

I tried 4 different implementations, which all failed due to the fact that the server side did not read the incoming data and the connection synchronization time. I will focus on two implementations that I have tried recently, and as far as I know, should work.

The client is divided into this request: there are 2 separate implementations, one of which is without a streamwriter.

textToSend = ENQUIRY + START_OF_TEXT + textToSend + END_OF_TEXT; // Send XML Request byte[] request = Encoding.UTF8.GetBytes(textToSend); using (DeflateStream streamOut = new DeflateStream(netStream, CompressionMode.Compress, true)) { //using (StreamWriter sw = new StreamWriter(streamOut)) //{ // sw.Write(textToSend); // sw.Flush(); streamOut.Write(request, 0, request.Length); streamOut.Flush(); //} } 

The server receives the request, and I do 1.) a quick look at the first character, if it matches what I expect
2.) I continue to read the rest.

The first read works correctly, and if I want to read the entire stream, that’s it. However, I only want to read the first character and evaluate it, and then continue in the LongReadStream method.

When I try to continue reading the stream, there is no data to read. I assume that data is lost during the first read, but I'm not sure how to determine this. All this code works correctly when I use a regular NetworkStream.

Here is the server side code.

 private void ProcessRequests() { // This method reads the first byte of data correctly and if I want to // I can read the entire request here. However, I want to leave // all that data until I want it below in my LongReadStream method. if (QuickReadStream(_netStream, receiveBuffer, 1) != ENQUIRY) { // Invalid Request, close connection clientIsFinished = true; _client.Client.Disconnect(true); _client.Close(); return; } while (!clientIsFinished) // Keep reading text until client sends END_TRANSMISSION { // Inside this method there is no data and the connection times out waiting for data receiveText = LongReadStream(_netStream, _client); // Continue talking with Client... } _client.Client.Shutdown(SocketShutdown.Both); _client.Client.Disconnect(true); _client.Close(); } private string LongReadStream(NetworkStream stream, TcpClient c) { bool foundEOT = false; StringBuilder sbFullText = new StringBuilder(); int readLength, totalBytesRead = 0; string currentReadText; c.ReceiveBufferSize = DEFAULT_BUFFERSIZE * 100; byte[] bigReadBuffer = new byte[c.ReceiveBufferSize]; while (!foundEOT) { using (var decompressStream = new DeflateStream(stream, CompressionMode.Decompress, true)) { //using (StreamReader sr = new StreamReader(decompressStream)) //{ //currentReadText = sr.ReadToEnd(); //} readLength = decompressStream.Read(bigReadBuffer, 0, c.ReceiveBufferSize); currentReadText = Encoding.UTF8.GetString(bigReadBuffer, 0, readLength); totalBytesRead += readLength; } sbFullText.Append(currentReadText); if (currentReadText.EndsWith(END_OF_TEXT)) { foundEOT = true; sbFullText.Length = sbFullText.Length - 1; } else { sbFullText.Append(currentReadText); } // Validate data code removed for simplicity } c.ReceiveBufferSize = DEFAULT_BUFFERSIZE; c.ReceiveTimeout = timeOutMilliseconds; return sbFullText.ToString(); } private string QuickReadStream(NetworkStream stream, byte[] receiveBuffer, int receiveBufferSize) { using (DeflateStream zippy = new DeflateStream(stream, CompressionMode.Decompress, true)) { int bytesIn = zippy.Read(receiveBuffer, 0, receiveBufferSize); var returnValue = Encoding.UTF8.GetString(receiveBuffer, 0, bytesIn); return returnValue; } } 

EDIT NetworkStream has a Socket property that has an Available property. MSDN is talking about an existing property.

Gets the amount of data received from the network, and available for reading.

Before calling below, 77 is available. After reading 1 byte, the value is 0.

 //receiveBufferSize = 1 int bytesIn = zippy.Read(receiveBuffer, 0, receiveBufferSize); 

There seems to be no documentation that DeflateStream consumes the entire base stream, and I don't know why it will do such a thing when there are explicit calls that need to be made to read certain numbers of bytes.

Does anyone know why this is happening, or if there is a way to save basic data for future reading? Based on this “function” and the previous article I read , indicating that DeflateStream must be closed in order to finish sending (the flash will not work) DeflateStreams may be limited in use for networks, especially if you want to resist DOS attacks by checking incoming data before taking the full stream.

Answer 1

Basically there are a few bugs with the code that I posted above. First, when I read data, I do nothing to make sure that all data is read. According to Microsoft documentation

A read operation reads as much data as is available, up to the number of bytes specified by the size parameter.

In my case, I was not convinced that my readings will receive all the data that I expected.

This can be done simply with this code.

 byte[] data= new byte[packageSize]; bytesRead = _netStream.Read(data, 0, packageSize); while (bytesRead < packageSize) bytesRead += _netStream.Read(data, bytesRead, packageSize - bytesRead); 

In addition to this problem, I had a fundamental problem using DeflateStream, namely: I should not use DeflateStream to write to the underlying NetworkStream. The correct approach is to first use DeflateStream to compress the data in ByteArray, and then send this ByteArray directly using NetworkStream.

Using this approach helped to correctly compress the data over the network, and the property read the data at the other end.

You can indicate that I should know the size of the data, and that’s true. Each call has an 8-byte header, which includes the size of the compressed data and the size of the data when it is uncompressed. Although I think that the second was completely unnecessary.

The code for this is here. Note that the compressSize variable fulfills 2 goals.

  int packageSize = streamIn.Read(sizeOfDataInBytes, 0, 4); while (packageSize!= 4) { packageSize+= streamIn.Read(sizeOfDataInBytes, packageSize, 4 - packageSize); } packageSize= BitConverter.ToInt32(sizeOfDataInBytes, 0); 

With this information, I can correctly use the code that I showed you, first of all, to get the contents completely.

Once I have a full compressed array of bytes, I can get the input as follows:

 var output = new MemoryStream(); using (var stream = new MemoryStream(bufferIn)) { using (var decompress = new DeflateStream(stream, CompressionMode.Decompress)) { decompress.CopyTo(output);; } } output.Position = 0; var unCompressedArray = output.ToArray(); output.Close(); output.Dispose(); return Encoding.UTF8.GetString(unCompressedArray); 

Answer 2

The main drawback that I can come up with when looking at your code is a possible misunderstanding of how network flow and compression work.

I think your code can work if you continue to work with one DeflateStream. However, you use it in your quick reading, and then create another one.

I will try to explain my reasoning with an example. Suppose you have 8 bytes of raw data that should be sent over the network in a compressed way. Now suppose for the argument that each byte (8 bits) of the source data will be compressed to 6 bits in compressed form. Now let's see what your code does.

You cannot read less than 1 byte from a network stream. You cannot take only 1 bit. You accept 1 byte, 2 bytes, or any number of bytes, but not a bit.

But if you want to get only 1 byte of source data, you need to read the first total byte of compressed data. However, there are only 6 bits of compressed data that represent the first byte of uncompressed data. The last 2 bits of the first byte are available for the second byte of the source data.

Now, if you cut the stream, the rest is 5 bytes in the network stream, which have no meaning and cannot be uncompressed.

The deflation algorithm is more complex than this, and therefore it makes sense if it does not allow you to stop reading from NetworkStream at one point and continue using the new DeflateStream from the middle. There is a decompression context that must be present in order to decompress the data to its original form. Once you get rid of the first DeflateStream in your quick read, this context has disappeared, you cannot continue.

So, to solve your problem, try to create only one DeflateStream and pass it to your functions, and then delete it.

Answer 3

It is overwhelmed in many ways.

• You assume that reading will read the exact number of bytes you want. However, it can read everything in bytes.
• DeflateStream has an internal buffer. It could not be otherwise: input bytes do not match 1: 1 for outputting bytes. There should be some internal buffering. You must use one such thread.
• Same problem with UTF-8: UTF-8 encoded strings cannot be divided by byte boundaries. Sometimes your data in Unicode will be distorted.
• Do not touch ReceiveBufferSize, it does not help in any way.
• You cannot reliably reset the descent stream, I think, because the output may be in partial byte position. You should probably develop a message frame format in which you add the compressed length as an uncompressed integer. Then send a compressed deflation stream after length. This is acceptable for decoding.

Solving these problems is not easy.

Since you seem to be managing the client and server, you should give up all this and not develop your own network protocol. Use a higher level mechanism such as web services, HTTP, protobuf. Everything is better than yours.