Download compressed content via HTTP using Python

Question

How can I use HTTP 1.1 compression when loading web pages using Python?

I am currently using the urllib built-in module to download web content. Reading the documentation, I could not find any information that really uses compression.

Is it already built into urllib or is there any other library that I can use?

Answer 1

httplib2 supports both deflate and gzip compression.

Example

import httplib2
h = httplib2.Http(".cache")
resp, content = h.request("http://example.org/", "GET")

Content is unpacked as needed.