Difficulty changing a Python method signature across Celery

Question

This is probably a Captain Obvious style question, but I feel like I'm missing something.

I inherrited a Python WSGI app that creates Celery tasks. That looks like this:

#web.py

from bottle import route, run, request, response, abort
import bottle
from project.tasks import process_request

@route('/')
def web():
    url = request.query_string
    process_request.delay(
        url=url,
        path=request.path,
        query=request.query_string,
        cookies=dict(request.cookies),
        headers=dict(request.headers),
        remote_addr=request.remote_addr,
    )

The process_request method looks like this:

#tasks.py
@celery.task
def process_request(url, path, query, cookies, headers, remote_addr):
    #does some stuff

All of the above works. I tried to add a timestamp to the process_request, so the call looks like this:

#web.py

#imports from above
from datetime import datetime

@route('/')
def web():
    url = request.query_string
    process_request.delay(
        url=url,
        path=request.path,
        query=request.query_string,
        cookies=dict(request.cookies),
        headers=dict(request.headers),
        remote_addr=request.remote_addr,
        s_time=str(datetime.now())
    )

And I changed the process_request method to look like:

#tasks.py
@celery.task
def process_request(url, path, query, cookies, headers, remote_addr, s_time):
    #does some stuff

That didn't work. Requests weren't being processed. I checked my celeryd log file, and found this:

[2013-03-26 16:19:03,330: ERROR/MainProcess] Task beacon.tasks.process_request[235f9fa8-1f10-4ee0-a1f9-e389021ea0ad] raised exception: TypeError('process_request() takes exactly 7 non-keyword arguments (6 given)',)
Traceback (most recent call last):
  File "/home/beacon/.virtualenvs/beacon/lib/python2.6/site-packages/celery/task/trace.py", line 228, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/beacon/.virtualenvs/beacon/lib/python2.6/site-packages/celery/task/trace.py", line 415, in __protected_call__
    return self.run(*args, **kwargs)
TypeError: process_request() takes exactly 7 non-keyword arguments (6 given)

I don't know what I'm doing wrong. Anyone have any ideas?

Answer 1

The problem here is that there are tasks queued which use the old method signature, being that without the timestamp variable. Without some more information, it is hard to determine your best course of action.

Are the tasks currently queued of vital importance? In my opinion, the easiest and cleanest way to solve this issue is to stop celery, empty the queue via Redis/RabbitMQ/ and then restart celery. Also, this should go without saying, make sure your task function accepts the new argument as a parameter ;)

Answer 2

1. add a new task, for example process_request_with_stime(), keep the old task process_request(), and replace the invocation in web().

2. deploy

3. after all the old queued jobs of task process_request() have been processed, remove the process_request() task, rename process_request_with_stime() to process_request()