Python Celery Task finished without backend

Question

I'm using Celery 3.0.12.

I have two queues: Q1, Q2.

In general I put the main task in Q1, which call then subtasks that go in Q2. I don't want to store any results for the subtasks. So my subtasks have the decorator @celery.task(ignore_results=True).

My main task should now wait until the subtask has finished. Because I write no results. I can't use: AsyncResult. Is there a way to wait in the main task to wait until the subtask finishes without storing the states to the backend. All my attempts with AsyncResults are not successfuel, (it relies on the backend). It seems also get() relies on the backend.

The whole story in code:

@celery.task(ignore_result=True)
def subtask():
   #Do something

@celery.task
def maintask():
    # Do something

    # Call subtask on Q2:
    res = subtask(options={'queue':'Q2'}).delay()

    # Need to wait till subtask finishes
    # NOT WORKING (DOES NEVER RETURN)
    res.get()

I'm monitoring the whole application with Celery Flower and I can see that subtask is successfuelly finishing. How can Celery detect that state? I browsed their code but couldn't find out how they do the detection.

Answer 1

My main task should now wait until the subtask has finished.

You should never wait for a subtask as this may lead to resource starvation and deadlock (all tasks waiting for another task, but no more workers to process them).

Instead you should use a callback to do additional actions after the subtask completes (see the Canvas guide in the Celery user guide).

I'm monitoring the whole application with Celery Flower and I can see that subtask is successfuelly finishing. How can Celery detect that state? I browsed their code but couldn't find out how they do the detection.

Flower and other monitors does not use results (task state), instead they use what we call events.

Event messages are emitted when certain actions occur in the worker, and this becomes a transient stream of messages. Processes can subscribe to certain events (or all of them) to monitor the cluster.

The events are separate from task states because,

• Events are not persistent (transient)

  Missing an event is not regarded as a critical failure.

• Complex fields are not serialized

  Events are for diagnostic and informational purposes, and should not be used to introspect task return values or exceptions for example, as only the repr() of these is stored to make sure monitors can be written in other languages, and big fields may be truncated to ensure faster transmission.