Celery tasks with unpickable arguments

Question

I'm trying to implement a messaging queue system to push operations to the AdWords API.

 class Call(object):
    @celery.task(filter=task_method)
    def MUTATE(self, operations):
        assert hasattr(self, '__throwaway_service')
        with self.__throwaway_service as sm:
            response = sm.mutate(operations)
        return response

since I cannot use the service instance as an argument (service has methods to send 'get' or 'mutate' requests to the API), i set it as external attribute and delete it when all the operations for that specific service are sent to the MQ.

'operations' is a list of dictionaries containing string or unicode keys and values.

I still get

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

Am I getting this error because the celery task is using an instance method at all? What is the best and pythonic way to implement this?

Answer 1

To run this MUTATE task, an instance of Call must be passed to queue. I suppose your Call, containing fancy methods such as __throwaway_service hardly is pickable. So you'd have to redesign your task, reconstructing self object in a task and applying original MUTATE afterwards.

Also, you might find this reciepe useful to find out what exactly object can't be pickled.

update

Is this pattern applicable:

@celery.task
def mutate(service_settings, operations):
    service = Service(**service_settings)
    return service.mutate(operations)

Answer 2

There is probably an object in operations. You need to process the object into dictionary or other native types before passing it to celery task.