Can iostreams read and compress gzipped files on the fly?

Question

I am reading a gzipped file using boost iostreams: The following works just fine:

namespace io = boost::iostreams; io::filtering_istream in; in.push(boost::iostreams::basic_gzip_decompressor<>()); in.push(io::file_source("test.gz")); stringstream ss; copy(in, ss); 

However, I do not want the memory to go into reading the entire gzip file into memory. I want to be able to read the file gradually.

For example, if I have an X data structure that is initialized from istream,

 X x; x.read(in); 

fails. Presumably, this is due to the fact that we may need to return characters to the stream if we do partial streams. Any ideas on whether iostreams supports this?

Answer 1

According to the iostream documentation, the type boost::io::filtering_istream comes from std::istream . That is, it should be possible to pass this everywhere, but std::istream& is expected. If you have runtime errors because you need unget() or putback() characters, you should look at the pback_size parameter, which indicates how many characters are returned maximum. I have not seen in the documentation what the default value is for this parameter.

If this does not solve your problem, can you accurately describe your problem? In appearance, it should work.

Answer 2

I think you need to write your own filter. For example, to read the .tar.gz file and output the contained files, I wrote something like

 //using namespace std; namespace io = boost::iostreams; struct tar_expander { tar_expander() : out(0), status(header) { } ~tar_expander() { delete out; } /* qualify filter */ typedef char char_type; struct category : io::input_filter_tag, io::multichar_tag { }; template<typename Source> void fetch_n(Source& src, std::streamsize n = block_size) { /* my utility */ .... } // Read up to n filtered characters into the buffer s, // returning the number of characters read or -1 for EOF. // Use src to access the unfiltered character sequence template<typename Source> std::streamsize read(Source& src, char* s, std::streamsize n) { fetch_n(src); const tar_header &h = cast_buf<tar_header>(); int r; if (status == header) { ... } std::ofstream *out; size_t fsize, stored; static const size_t block_size = 512; std::vector<char> buf; enum { header, store_file, archive_end } status; } } 

My read(Source &...) function gets unzipped text when called. To use a filter:

 ifstream file("/home/..../resample-1.8.1.tar.gz", ios_base::in | ios_base::binary); io::filtering_streambuf<io::input> in; in.push(tar_expander()); in.push(io::gzip_decompressor()); in.push(file); io::copy(in, cout);