Downloading compressed content over HTTP using Python
-
19-09-2019 - |
Question
How do I take advantage of HTTP 1.1's compression when downloading web pages using Python?
I am currently using the built-in urllib module for downloading web content. Reading through the documentation I couldn't find any information that is indeed using compression.
Is it already built-in into urllib or is there another library that I can use?
Solution
httplib2 supports 'deflate' and 'gzip' compression.
Example
import httplib2
h = httplib2.Http(".cache")
resp, content = h.request("http://example.org/", "GET")
The content is decompressed as necessary.
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow