Error with urlopen: new-line character seen in unquoted field

Question

I am using urllib.urlopen with Python 2.7 to read csv files located on an external webserver:

# Try & Except statements removed for clarity
import urllib
import csv
url = ...
csv_file = urllib.urlopen(url)
for row in csv.reader(csv_file):
    do_something()

All 100+ files can be read fine, except one that has been updated recently and that returns:

Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

The file is accessible here. According to my text editor, its mode is Mac (CR), as opposed to Windows (CRLF) for the other files.

I found that based on this thread, python urlopen will handle correctly all formats of newlines. Therefore, the problem is likely to come from somewhere else. I have no clue though. The file opens fine with all my text editors and my speadsheet editors.

Does any one have any idea how to diagnose the problem ?

* EDIT *

The creator of the file informed me by email that I was not the only one to experience such issues. Therefore, he decided to make it again. The code above now works fine again. Unfortunately, using a new file also means that the issue can no longer be reproduced, and the solutions tested properly.

Before closing the question, I want to thank all the stackers who dedicated some of their time to figure out a solution and post it here.

Answer 1

It might be a corrupt .csv file? Otherwise, this code runs perfectly.

#!/usr/bin/python

import urllib
import csv

url = "http://www.football-data.co.uk/mmz4281/1213/I1.csv"
csv_file = urllib.urlopen(url)

for row in csv.reader(csv_file):
  print row

Credits to J.F. Sebastian for the .csv file.

Altough, you might want to consider sharing the specific .csv file with us? So we can try to re-create the error.

Answer 2

The following code runs without any error:

#!/usr/bin/env python
import csv
import urllib2

r = urllib2.urlopen('http://www.football-data.co.uk/mmz4281/1213/I1.csv')
for row in csv.reader(r):
    print row

Answer 3

I was having the same problem with a downloaded csv.

I know the fix would be to use open with 'rU'. But I would rather not have to save the file to disk, just to open back up into a variable. That seems unnecessary.

file = open(filepath,'rU')
mydata = csv.reader(file)

So if someone has a better solution that would be nice. Stackoverflow links that got me this far:

CSV new-line character seen in unquoted field error

Open the file in universal-newline mode using the CSV Django module



I found what I actually wanted with stringIO, or cStringIO, or io:

Using Python, how do I to read/write data in memory like I would with a file?

I ended up getting io working,

import csv
import urllib2
import io
# warning its a 20MB csv
url = 'http://poweredgec.com/latest_poweredge-11g.csv'
urlRead = urllib2.urlopen(url).read()
ramFile = io.open(urlRead, mode='w')
openRamFile = open(ramFile, 'rU')
csvCurrent = csv.reader(openRamFile)
csvTuple = map(tuple, csvCurrent)

print csvTuple