Error while reading compressed CSV file from URL with pandas.read_csv

Question

I'm trying to use pandas.read_csv to read bz2-compressed CSV files from a URL (in this case, a GitHub raw URL) from within an IPython notebook, but I get the following error:

Python cannot read bz2 from open file handle.

I've done some research and I can't seem to figure out what's wrong. I've tried manually decompressing the bz2 files, and I know they aren't corrupted or otherwise bad, and I also know that the URLs are correctly formatted - if I enter them into my browser, it downloads the file correctly.

Here's the code from my IPython notebook:

import pandas, bz2, urllib2
datafiles = {}
for filename in csvData['Filename']:
    csvUrl = 'https://raw.github.com/hawkw/traverse/master/data/' + filename
    try:
        datafiles[filename] = pandas.read_csv(csvUrl, compression='bz2')
        print (datafiles[filename])
    except Exception as e:
        print("Caught error \"{error}\" at {url}".format(error=e, url=csvUrl))

And the output:

Caught error "Python cannot read bz2 from open file handle" at https://raw.github.com/hawkw/traverse/master/data/-Users-hawk-_2014-02-05.csv.bz2
Caught error "Python cannot read bz2 from open file handle" at https://raw.github.com/hawkw/traverse/master/data/-Users-Owner_2014-02-05.csv.bz2
Caught error "Python cannot read bz2 from open file handle" at https://raw.github.com/hawkw/traverse/master/data/-Users-will_2014-02-05.csv.bz2
Caught error "Python cannot read bz2 from open file handle" at https://raw.github.com/hawkw/traverse/master/data/-Users-hawk_2014-02-06.csv.bz2
Caught error "Python cannot read bz2 from open file handle" at https://raw.github.com/hawkw/traverse/master/data/-Users-hawk-Documents_2014-02-06.csv.bz2
Caught error "Python cannot read bz2 from open file handle" at https://raw.github.com/hawkw/traverse/master/data/-home-w-weismanm_2014-02-05.csv.bz2
Caught error "Python cannot read bz2 from open file handle" at https://raw.github.com/hawkw/traverse/master/data/-home-w-weismanm_2014-02-06.csv.bz2

Anybody know what I'm doing wrong?

Edit: I tried opening the file using urllib2 like this, as @edchum suggested:

datafiles = {}

for filename in csvData['Filename']:

    url = 'https://raw.github.com/hawkw/traverse/master/data/' + filename

    try:
        response = urllib2.urlopen(url)
    except HTTPError as e:
        print ("Caught HTTPError", e)
    else:
        try:
            datafiles[filename] = pandas.read_csv(response, compression='bz2')
            print (datafiles[filename])
        except Exception as e:
            print("Caught error \"{0}\" at {1}".format(e,url))

But it still didn't work, failing with the same error. As a side note, pandas.read_csv() says that it can open files from URLs in the docs.

Answer 1

If you absolutely can't simply download the files for some reason, and have to keep pulling them from a remote source (which isn't very neighbourly), then you could do it all in-memory if needed:

>>> import pandas as pd
>>> import io
>>> import urllib2
>>> import bz2
>>> 
>>> url = "https://github.com/hawkw/traverse/blob/master/data/-Users-hawk_2014-02-06.csv.bz2?raw=true"
>>> raw_data = urllib2.urlopen(url).read()
>>> data = bz2.decompress(raw_data)
>>> df = pd.read_csv(io.BytesIO(data))
>>> df.head()
                                                path  st_mode    st_ino  \
0  /Users/hawk/Library/minecraft/bin/minecraft/mo...    33261  59612469   
1  /Users/hawk/Library/Application Support/Google...    16832  91818463   
2  /Users/hawk/Library/Caches/Metadata/Safari/His...    33188  95398522   
3  /Users/hawk/Documents/Minecraft1.6.4/assets/so...    33188  90620503   
4  /Users/hawk/Library/Caches/Metadata/Safari/His...    33188  96129272   

     st_dev  st_nlink  st_uid  st_gid  st_size    st_atime    st_mtime  \
0  16777219         1     501      20     2626  1370201925  1366983504   
1  16777219         3     501      20      102  1391697388  1384638452   
2  16777219         1     501      20    36758  1389032348  1389032363   
3  16777219         1     501      20    12129  1387000073  1384114141   
4  16777219         1     501      20      170  1390545751  1390545751   

     st_ctime  
0  1368736019  
1  1384638452  
2  1389032363  
3  1384114141  
4  1390545751  

[5 rows x 11 columns]

where you'd want to pass the appropriate encoding parameter.