How to make Apache mod_deflate and Transfer-encoding: Chunked work together?

Question

I am trying to use the bigpipe concept on our website. This means trying to send the answer in pieces, rather than sending it as a whole, so that the user feels that the page is running fast. I succeed using the flushBuffer method for the response object in java. But now, when I try to compress the content with the apache mod_deflate module, the fragment is lost.

Here is the configuration from apache used to compress content

**

Start mod_deflate config

DeflateBufferSize 100 AddOutputFilterByType DEFLATE text/html text/plain text/xml text/javascript BrowserMatch ^Mozilla/4 gzip-only-text/html BrowserMatch ^Mozilla/4\.0[678] no-gzip BrowserMatch \bMSIE !no-gzip !gzip-only-text/html DeflateFilterNote Input input_info DeflateFilterNote Output output_info DeflateFilterNote Ratio ratio_info LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate CustomLog /var/log/httpd/deflate_log deflate 

End mod_deflate config **

Here is the response header when deflate is enabled in apache

Connection: Keep-Alive
Content-Encoding: GZIP
Content-Length: 7916
Content-Type: Text / html; encoding = UTF-8
Date: Fri, 27 Jan 2012 20:11:11 GMT
Keep-Alive: timeout = 300, max = 3997
Server: apache
Vary: Accept-Encoding

Answer header when deflation is disabled in apache

Connection: Keep-Alive
Content-Type: Text / html; encoding = UTF-8
Date: Fri, 27 Jan 2012 20:21:14 GMT
Keep-Alive: timeout = 300, max = 3997
Server: Apache / 2.2.3 (CentOS)
Transmission Encoding: chunked

As you can see above, two headers only work if compression is disabled. I searched the Internet about this, and people suggested reducing the value of DeflateBufferSize . I reduced the value to 100 bytes, as you can see in my apache configuration, but this has not solved the problem yet. DeflateBufferSize set to 100 bytes means that the response is buffered in apache until 100 bytes are received and then compressed.

I watched the mod_gzip module, which was linked with old apache 1.3, and this module has the following directive, which allows you to download fragmented content.

mod_gzip_dechunk Yes

Does anyone know about such a directive in mod_deflate bundled with apache 2.x?

Or does anyone know how to compress tagged content?

Answer 1

I actually found a solution. Each time I created a new GZipOutputStream object to clear different pieces. Instead, you should create one object only GZipOutputStream, and then use this object to compress all fragments of the response. I also put a wrapper around the GZipOutputStream. Here is the wrapper I got from googling.

 public class GZIPFlushableOutputStream extends GZIPOutputStream { public GZIPFlushableOutputStream(final OutputStream out) throws IOException { // Using Deflater with nowrap == true will ommit headers and trailers super(out); } private static final byte[] EMPTYBYTEARRAY = new byte[0]; /** * Insure all remaining data will be output. */ public void flush() throws IOException { /** * Now this is tricky: We force the Deflater to flush its data by * switching compression level. As yet, a perplexingly simple workaround * for * * http://developer.java.sun.com/developer/bugParade/bugs/42557 43.html */ def.setInput(EMPTYBYTEARRAY, 0, 0); def.setLevel(Deflater.NO_COMPRESSION); deflate(); def.setLevel(Deflater.DEFAULT_COMPRESSION); deflate(); out.flush(); } } 

Answer 2

My understanding is that you need a “whole” file to compress it. You can either send it to pieces, or send it in a compressed way. The mod_gzip_dechunk option no longer exists - see mod_deflate documentation .