Python: Check the url request per hour

Question

I am accessing a api and extracting a json but I want to make sure I stay within the hourly request limit, what would be the best way to do this?

This where I make the request:

# return the json
def returnJSONQuestion(id):
    url = 'http://someApi.com?index_id={0}&output=json'
    format_url = url.format(id)
    try:
        urlobject = urllib2.urlopen(format_url)
        jsondata = json.loads(urlobject.read().decode("utf-8"))
        print jsondata
        shortRandomSleep()
    except urllib2.URLError, e:
        print e.reason
    except(json.decoder.JSONDecodeError,ValueError):
        print 'Decode JSON has failed'
    return jsondata

Answer 1

I usually use a cheap hack where I make the script run every other minute by checking the current time. This is the general form of the function:

def minuteMod(x, p=0):
    import datetime
    minute = datetime.datetime.now() + datetime.timedelta(seconds=15)
    minute = int(datetime.datetime.strftime(minute, "%M"))
    if minute % x == p:
        return True
    return False

p is the remainder here and has a default value of 0 so no particular need to pass in the second argument.

So basically, if you want your script to run only every other minute, you use it like this:

def returnJSONQuestion(id):

    if not minuteMod(2):
        return None or ''

    # rest of the code

This will stop the request if the current minute is not even. Considering this is not the best way to do things, you can use this function to cache results (depending on if this is allowed). So basically, you would do something like this:

def returnJSONQuestion(id):

    if minuteMod(3): # current minute is a factor of 3
        return jsonFromCache # open a file and output cached contents
    else:
        url = 'http://...'
        storeJSONToFile(url)
        return json

Answer 2

You could use a token bucket algorithm, something like this: http://code.activestate.com/recipes/511490/

Have tokens added to the bucket at the rate the API allows you to make requests, and take a token from the bucket each time you make a request.