UTF-8 minus sign rejected in python command-line arguments

Question

I am running python 2.6 on Ubuntu Lucent and having trouble getting the minus sign in negative command-line arguments to be interpreted properly, especially when the call to the script is initiated through the OS via Rails (using backquotes). In particular, the minus sign seems to be coming in as UTF-8.

When command-line arguments are interpreted manually, as in:

lng = float(sys.argv[4])

it triggers the error:

ValueError: invalid literal for float(): ‐122.768

As a hack, I can get around this by matching on the first three bytes as '\xe2', '\x80', and '\x90', and replacing them with my own negative sign.

When command-line arguments are interpreted through argparse (ver. 1.2.1), as in:

parser.add_argument('--coords', metavar='Coord', dest='coordinates', type=float, nargs=3, help='Latitude, Longitude, and Altitude')

it triggers the error:

sC.py: error: argument --coords: invalid float value: '\xe2\x80\x90122.76838'

Any help would be appreciated!

Answer 1

You might have to use your hack and tell argparse to expect a string.

As far as Python, your system, and RoR are concerned - and ― aren't related in any way. If you want to solve this problem (instead of hack it) you have go up to the rails code, and see where it gets its data from. Somewhere along the line fancy output was important.

Answer 2

Your input data contains a Unicode character that isn't the standard ascii hyphen.

import unicodedata as ud
data = '\xe2\x80\x90122.76838'
unicode_data = data.decode('utf8')
print repr(ud.name(unicode_data[0]))
print repr(ud.name(u'-')) # An ascii hyphen

Output:

'HYPHEN'
'HYPHEN-MINUS'

While they may look the same when printed, they are not. Restrict or sanitize the input.

print float(unicode_data.replace(u'\N{HYPHEN}',u'-'))

Output:

-122.76838