Using a zlib filter with a pair of slots

Question

For some reason, the zlib.deflate filter does not seem to work with socket pairs generated by stream_socket_pair() . All that can be read from the second socket is the double-byte zlib header, and everything after it is NULL.

Example:

 <?php list($in, $out) = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, STREAM_IPPROTO_IP); $params = array('level' => 6, 'window' => 15, 'memory' => 9); stream_filter_append($in, 'zlib.deflate', STREAM_FILTER_WRITE, $params); stream_set_blocking($in, 0); stream_set_blocking($out, 0); fwrite($in, 'Some big long string.'); $compressed = fread($out, 1024); var_dump($compressed); fwrite($in, 'Some big long string, take two.'); $compressed = fread($out, 1024); var_dump($compressed); fwrite($in, 'Some big long string - third time is the charm?'); $compressed = fread($out, 1024); var_dump($compressed); 

Conclusion:

 string(2) "x " string(0) "" string(0) "" 

If I comment on the call to stream_filter_append() , the stream write / read function works correctly, all data will be completely reset three times, and if I direct the stream of zlib filters to a file, and not through a steam socket, the compressed data is written correctly. Thus, both parts function correctly separately, but not together. Is this a PHP bug that I should report, or a bug on my part?

This question is forked from solving this related question .

Answer 1

Looking through the C source code , the problem is that the filter always allows zlib deflate() determine how much data is accumulated before generating compressed output. The deflation filter does not create a new data bucket for transmission if deflate() does not output some data (see line 235) or the flag bit PSFS_FLAG_FLUSH_CLOSE (line 250). This is why you only see header bytes until you close $in ; the first call to deflate() outputs two bytes of the header, so data->strm.avail_out is 2, and a new bucket is created for these two bytes.

Note that fflush() does not work due to a known problem with the zlib filter. See: Error # 48725 Support for cleaning in the zlib stream .

Unfortunately, it seems like this is not so nice. I started writing a filter in PHP by expanding php_user_filter , but quickly ran into the problem that php_user_filter did not set flag bits, but only flags & PSFS_FLAG_FLUSH_CLOSE (the fourth parameter for the filter() method, a logical argument usually called $closing ). You will need to change the C sources yourself to fix the error # 48725. Alternatively, rewrite it.

Personally, I would think about rewriting it, because there are apparently several problems with the eyebrow with the code:

• status = deflate(&(data->strm), flags & PSFS_FLAG_FLUSH_CLOSE ? Z_FULL_FLUSH : (flags & PSFS_FLAG_FLUSH_INC ? Z_SYNC_FLUSH : Z_NO_FLUSH)); seems strange, because when I write, I don’t know why flags will be anything other than PSFS_FLAG_NORMAL . Is it possible to write and hide at the same time? In any case, flag processing should be performed outside the while through the "in" command console, for example, how PSFS_FLAG_FLUSH_CLOSE processed outside this loop.

• Line 221, from memcpy to data->strm.next_in , seems to ignore the fact that data->strm.avail_in may be non-zero, so compressed output may skip some write data. See, for example, the following text from the zlib manual:

  If not all input can be processed (because there is not enough space in the output buffer), next_in and avail_in , and processing will resume at that moment for the next deflate() call.

  In other words, it is possible that avail_in nonzero.

• The if on line 235, if (data->strm.avail_out < data->outbuf_len) should probably be if (data->strm.avail_out) or, possibly, if (data->strm.avail_out > 2) .
• I'm not sure why *bytes_consumed = consumed; not *bytes_consumed += consumed; . In the sample streams, http://www.php.net/manual/en/function.stream-filter-register.php everyone uses += to update $consumed .

EDIT: *bytes_consumed = consumed; true. Standard filter implementations use = instead of += to update the size_t value pointed to by the fifth parameter. Also, although $consumed += ... on the PHP side effectively translates to += on size_t (see Lines 206 and 231 ext/standard/user_filters.c ), the native filter function is called using a NULL pointer or a pointer to size_t , set to 0 for the fifth argument (see lines 361 and 452 main/streams/filter.c ).

Answer 2

I worked on PHP source code and found a fix.

To understand what was going on, I traced the code during

 .... for ($i = 0 ; $i < 3 ; $i++) { fwrite($s[0], ...); fread($s[1], ...); fflush($s[0], ...); fread($s[1], ...); } 

and I found that the deflate function is never called with the Z_SYNC_FLUSH flag, because new data is not present in the backets_in brigade.

My fix is ​​to control the checkbox ( PSFS_FLAG_FLUSH_INC set AND iterations are not performed in the case of the deflation function), expanding

 if (flags & PSFS_FLAG_FLUSH_CLOSE) { 

management of FLUSH_INC too:

 if (flags & PSFS_FLAG_FLUSH_CLOSE || (flags & PSFS_FLAG_FLUSH_INC && to_be_flushed)) { 

This downloadable patch is for the debian squeeze version of PHP, but the current version of the git file is closer to it, so I suppose it's easy to transfer the fix (a few lines).

If any side effect occurs, contact me.

Answer 3

You need to close the stream after writing to clear it before the data comes from reading.

 list($in, $out) = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, STREAM_IPPROTO_IP); $params = array('level' => 6, 'window' => 15, 'memory' => 9); stream_filter_append($out, 'zlib.deflate', STREAM_FILTER_WRITE, $params); stream_set_blocking($out, 0); stream_set_blocking($in, 0); fwrite($out, 'Some big long string.'); fclose($out); $compressed = fread($in, 1024); echo "Compressed:" . bin2hex($compressed) . "<br>\n"; list($in, $out) = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, STREAM_IPPROTO_IP); $params = array('level' => 6, 'window' => 15, 'memory' => 9); stream_filter_append($out, 'zlib.deflate', STREAM_FILTER_WRITE, $params); stream_set_blocking($out, 0); stream_set_blocking($in, 0); fwrite($out, 'Some big long string, take two.'); fclose($out); $compressed = fread($in, 1024); echo "Compressed:" . bin2hex($compressed) . "<br>\n"; list($in, $out) = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, STREAM_IPPROTO_IP); $params = array('level' => 6, 'window' => 15, 'memory' => 9); stream_filter_append($out, 'zlib.deflate', STREAM_FILTER_WRITE, $params); stream_set_blocking($out, 0); stream_set_blocking($in, 0); fwrite($out, 'Some big long string - third time is the charm?'); fclose($out); $compressed = fread($in, 1024); echo "Compressed:" . bin2hex($compressed) . "<br>\n"; 

This gives: Compressed: 789c0bcecf4d5548ca4c57c8c9cf4b57282e29cacc4bd70300532b079c Compressed: 789c0bcecf4d5548ca4c57c8c9cf4b57282e29cacc4bd7512849cc4e552829cfd70300b1b50b07 Compressed: 789c0bcecf4d5548ca4c57c8c9cf4b57282e29ca0452ba0a25199945290a259940c9cc62202f55213923b128d71e008e4c108c

I also switched $ in and $ out, because the entry in $ confused me.