Proper way to deal with variations in JSON serialization

Question

I have a web service that uses Python's SimpleJSON to serialize JSON, and a javascript/ client that uses Google's Visualization API. When I try to read in the JSON response using Google Data Table's Query method, I am getting a "invalid label" error.

I noticed that Google spreadsheet outputs JSON without quotes around the object keys. I tried reading in JSON without the quotes and that works. I was wondering what was the best way to get SimpleJSON output to be read into Google datable using

query = new google.visualization.Query("http://www.myuri.com/api/").

I could use a regex to remove the quotes, but that seems sloppy. The javascript JSON parsing libraries I've tried won't read in JSON syntax without quotes around the object keys.

Here's some good background reading re: quotes around object keys:

http://simonwillison.net/2006/Oct/11/json/.

Answer 1

Are you certain the Google API is expecting JSON? In my experience Google's APIs tend not to be massively broken in a manner you're describing -- it could be that they're actually expecting a different format that merely resembles JSON.

---

Further poking around reveals instructions for retrieving data in the format Google expects:

For example, to get the dataSourceUrl from a Google Spreadsheet, do the following:

1. In your spreadsheet, select the range of cells.
2. Select 'Insert' and then 'Gadget' from the menu.
3. Open the gadget's menu by clicking on the top-right selector.
4. Select menu option 'Get data source URL'.

I did this and opened the URL in my browser. The data it was returning was certainly not JSON:

google.visualization.Query.setResponse(
{requestId:'0',status:'ok',signature:'1464883469881501252',
table:{cols: [{id:'A',label:'',type:'t',pattern:''},
{id:'B',label:'',type:'t',pattern:''}],
rows: [[{v:'a'},{v:'h'}],[{v:'b'},{v:'i'}],[{v:'c'},{v:'j'}],[{v:'d'},{v:'k'}],[{v:'e'},{v:'l'}],[{v:'f'},{v:'m'}],[{v:'g'},{v:'n'}]]}});

It looks like the result is intended to be directly executed by the browser. Try modifying your code to do something like this:

# old
return simplejson.dumps ({"requestId": 1, "status": "ok", ...})

# new
json = simplejson.dumps ({"requestId": 1, "status": "ok", ...})
return "google.visualization.Query.setResponse(%r);" % json

Answer 2

The "invalid label" error is usually due to a blind eval() on the JSON string, resulting in property names being mistaken as labels (because they have the same syntax -- "foo:").

eval("{ foo: 42, bar: 43 }"); // Results in invalid label

The quick remedy is to make sure your JSON string has parenthesis enclosing the curly braces:

eval("({ foo: 42, bar: 43 })"); // Works

Try enclosing your JSON string in parenthesis to see if the "invalid label" error goes away.

Answer 3

As it turns out :mod:json would also choke at strings in single quotes. This will sort things out though:

Parse JavaScript object as JSON in python:

solution:

>>> from re import sub
>>> import json
>>> js = "{ a: 'a' }"
>>> json.loads(sub("'", '"', sub('\s(\w+):', r' "\1":', js)))
{u'a': u'a'}

Edit: (edge cases reviewed)

So it was brought up that the suggested solution would not cope with all cases and specifically with something like

e.g. {foo: "a sentence: right here!"} will get changed to {"foo": "a "sentence": right here!"}
– Jason S Apr 12 at 18:03

To resolve that we simply need to ensure that we are in fact working with a key and not simply a colon in a string so we do a little look behind magic to hint at a comma(,) or a curly brace({) presence to ensure we have it proper, like so:

colon in string:

>>> js = "{foo: 'a sentence: right here!'}"
>>> json.loads(sub("'", '"', sub('(?<={|,)\s*(\w+):', r' "\1":', js)))
{u'foo': u'a sentence: right here!'}

Which of course is the same as doing:

>>> js = "{foo: 'a sentence: right here!'}"
>>> json.loads(sub('(?<={|,)\s*(\w+):', r' "\1":', js).replace("'",'"'))
{u'foo': u'a sentence: right here!'} 

But then I pointed out that this is not the only flaw because what about quotes:

If we are also concerned about escaped quotes we will have to be slightly more specific as to what constitutes a string. The first quote will follow either a curly brace({) a space(\s) or a colon(:) while the last matching quote will come before either a comma(,) or a closing curly brace(}) then we can consider everything in between as part of the same string, like so:

additional quotes in string:

>>> js = "{foo: 'a sentence: it\'s right here!'}"
>>> json.loads(
...     sub("(?<=\s|{|:)'(.*?)'(?=,|})", 
...         r'"\1"', 
...         sub('(?<={|,)\s*(\w+):', r' "\1":', js))
...     )
{u'foo': u"a sentence: it's right here!"}

Watch this space as more edge cases are revealed and solved. Can you spot another?

Or for something more complex perhaps, a real world example as returned by npm view:

From:

{ name: 'chuck',
      description: 'Chuck Norris joke dispenser.',
      'dist-tags': { latest: '0.0.3' },
      versions: '0.0.3',
      maintainers: 'qard ',
      time: { '0.0.3': '2011-08-19T22:00:54.744Z' },
      author: 'Stephen Belanger ',
      repository: 
           { type: 'git',
             url: 'git://github.com/qard/chuck.git' },
          version: '0.0.3',
          dependencies: { 'coffee-script': '>= 1.1.1' },
          keywords: 
               [ 'chuck',
                 'norris',
                 'jokes',
                 'funny',
                 'fun' ],
              bin: { chuck: './bin/chuck' },
              main: 'index',
              engines: { node: '>= 0.4.1 < 0.5.0' },
              devDependencies: {},
              dist: 
                   { shasum: '3af700056794400218f99b7da1170a4343f355ec',
                     tarball: 'http://registry.npmjs.org/chuck/-/chuck-0.0.3.tgz' },
                  scripts: {},
                  directories: {},
                  optionalDependencies: {} }

To:

{u'author': u'Stephen Belanger ',
     u'bin': {u'chuck': u'./bin/chuck'},
     u'dependencies': {u'coffee-script': u'>= 1.1.1'},
     u'description': u'Chuck Norris joke dispenser.',
     u'devDependencies': {},
     u'directories': {},
     u'dist': {u'shasum': u'3af700056794400218f99b7da1170a4343f355ec',
      u'tarball': u'http://registry.npmjs.org/chuck/-/chuck-0.0.3.tgz'},
     u'dist-tags': {u'latest': u'0.0.3'},
     u'engines': {u'node': u'>= 0.4.1 < 0.5.0'},
     u'keywords': [u'chuck', u'norris', u'jokes', u'funny', u'fun'],
     u'main': u'index',
     u'maintainers': u'qard ',
     u'name': u'chuck',
     u'optionalDependencies': {},
     u'repository': {u'type': u'git', u'url': u'git://github.com/qard/chuck.git'},
     u'scripts': {},
     u'time': {u'0.0.3': u'2011-08-19T22:00:54.744Z'},
     u'version': u'0.0.3',
     u'versions': u'0.0.3'}

Works for me =)

nJoy!