CURL and urlopen giving conflicting results

Question

So I'm retrieving some JSON data from a website and using the standard urlopen() from urllib2:

Code:

url = 'http://api.nytimes.com/svc/politics/v3/us/legislative/congress/senate/votes/2011-12-14/2011-12-14.json?api-key=%s' % (api_key,)
print urlopen(url).read()  

Result:

{
   "status":"OK",
   "copyright":"Copyright (c) 2011 The New York Times Company. All Rights Reserved.",
   "results": {
      "chamber": "Senate",
      "start_date": "2011-12-14",
      "end_date": "2011-12-14",
      "num_results": "0",
      "votes": [
                 ]
    }
}

This result is wrong. If I enter the url manually into my browser:

Entering:

http://api.nytimes.com/svc/politics/v3/us/legislative/congress/senate/votes/2011-12-14/2011-12-14.json?api-key=a55a8988c8102a532221d2d465ca010a:1:65328279

I get:

{
   "status":"OK",
   "copyright":"Copyright (c) 2011 The New York Times Company. All Rights Reserved.",
   "results": {
      "chamber": "Senate",
      "start_date": "2011-12-14",
      "end_date": "2011-12-14",
      "num_results": "1",
  "votes": [
            {
       "congress": "112",
       "session": "1",
       "roll_call": "228",
"vote_uri":"http://api.nytimes.com/svc/politics/v3/us/legislative/congress/112/senate/sessions/1/votes/228.json",
       "bill_number": "S.J.Res.24",
       "question": "On the Joint Resolution", 
       "description": "Joint resolution proposing a balanced budget amendment to the Constitution of the United States.",
       "vote_type": "1/2",
       "date": "2011-12-14",
       "time": "11:09:00",
       "result": "Joint Resolution Defeated",
       "democratic": {
         "yes": "20",
         "no": "31",
         "present": "0",
         "not_voting": "0",
         "majority_position": "No"
        },
       "republican": {
         "yes": "1",
         "no": "46",
         "present": "0",
         "not_voting": "0",
         "majority_position": "No"
        },
       "independent": {
         "yes": "0",
         "no": "2",
         "present": "0",
         "not_voting": "0"
        },
       "total": {
         "yes": "21",
         "no": "79",
         "present": "0",
         "not_voting": "0"
        }
      }                 ]
}

}

Also, when I retrieve the data using CURL I get:

$curl http://api.nytimes.com/svc/politics/v3/us/legislative/congress/senate/votes/2011-12-14/2011-12-14.json?api-key=a55a8988c8102a532221d2d465ca010a:1:65328279
{
   "status":"OK",
   "copyright":"Copyright (c) 2011 The New York Times Company. All Rights Reserved.",
   "results": {
      "chamber": "Senate",
      "start_date": "2011-12-14",
      "end_date": "2011-12-14",
      "num_results": "2",
       "votes": [
            {
       "congress": "112",
       "session": "1",
       "roll_call": "229",
       "vote_uri": "http://api.nytimes.com/svc/politics/v3/us/legislative/congress/112/senate/sessions/1/votes/229.json",
       "bill_number": "S.J.Res.10",
       "question": "On the Joint Resolution", 
       "description": "Joint resolution proposing a balanced budget amendment to the Constitution of the United States.",
       "vote_type": "2/3",
       "date": "2011-12-14",
       "time": "11:39:00",
       "result": "Joint Resolution Defeated",
       "democratic": {
         "yes": "0",
         "no": "51",
         "present": "0",
         "not_voting": "0",
         "majority_position": "No"
        },
       "republican": {
         "yes": "47",
         "no": "0",
         "present": "0",
         "not_voting": "0",
         "majority_position": "Yes"
        },
       "independent": {
         "yes": "0",
         "no": "2",
         "present": "0",
         "not_voting": "0"
        },
       "total": {
         "yes": "47",
         "no": "53",
         "present": "0",
         "not_voting": "0"
        }
      },                {
       "congress": "112",
       "session": "1",
       "roll_call": "228",
       "vote_uri": "http://api.nytimes.com/svc/politics/v3/us/legislative/congress/112/senate/sessions/1/votes/228.json",
       "bill_number": "S.J.Res.24",
       "question": "On the Joint Resolution", 
       "description": "Joint resolution proposing a balanced budget amendment to the Constitution of the United States.",
       "vote_type": "1/2",
       "date": "2011-12-14",
       "time": "11:09:00",
       "result": "Joint Resolution Defeated",
       "democratic": {
         "yes": "20",
         "no": "31",
         "present": "0",
         "not_voting": "0",
         "majority_position": "No"
        },
       "republican": {
         "yes": "1",
         "no": "46",
         "present": "0",
         "not_voting": "0",
         "majority_position": "No"
        },
       "independent": {
         "yes": "0",
         "no": "2",
         "present": "0",
         "not_voting": "0"
        },
       "total": {
         "yes": "21",
         "no": "79",
         "present": "0",
         "not_voting": "0"
        }
      }                 ]
}

}

So that's 3 different results for the same web request. What's the issue here? Is there some caching mechanism that I'm overlooking or am I missing something?

Answer 1

It looks like you have to have 'Accept-Encoding' header set in your request.

Just in case, you can set like this:

request = urllib2.Request(your_url)
request.add_header('Accept-Encoding',whatever_in_this_case)
opener = urllib2.build_opener()
print opener.open(request).read()