Python 3 Urlopen vs Urlretreive

Question

I am working on a script to download and process historical stock prices. When I used urllib.request.urlopen I got a strange prefix of text in every file (b'\xef\xbb\xbf) that was not present when I used urllib.request.urlretrieve, nor present when I typed the url into a browser (Firefox). So I have an answer but I don't know why it was causing a problem in the first place. I suspect it may be because I forced it to be a string, but I don't know why that is or how I would work around that (other than to use urlretrieve instead). The code is below. The relevant line is line 11. The commented code after is when I was using orlopen.

    #download a bunch of historical stock quotes from google finance

import urllib.request
symbolarray = []
symbolfile = open("symbols.txt")
for line in symbolfile:
    symbolarray.append(line.strip())
symbolfile.close()

for symbol in symbolarray:
    page = urllib.request.urlretrieve("http://www.google.com/finance/historical?q=NYSE:"+symbol+"&output=csv",symbol+".csv")
    #datafile = open(symbol+".csv","w")
    #datafile.write(str(page.read()))
    #datafile.close()

Answer 1

0xEF,0xBB,0xBF is the BOM for utf-8. It signifies that this is a utf-8 encoded string. I'm guessing that if you use wireshark you'll see that it was there all along. It's just that most programs ignore it.

Instead of str(page.read()) you should try page.read().decode('utf-8-sig') if you want to remove the BOM. If you want to keep it you can decode just with 'utf-8'.