Celery task with a time_start attribute in 1970

Question

An inspection of currently running Celery tasks reveals a weird time_start timestamp:

>> celery.app.control.inspect().active()
{u'celery@worker.hostname': [{u'acknowledged': True,
   u'args': u'(...,)',
   u'delivery_info': {u'exchange': u'celery',
    u'priority': 0,
    u'redelivered': None,
    u'routing_key': u'celery'},
   u'hostname': u'celery@worker.hostname',
   u'id': u'3d92fdfd-524e-4ba1-98cb-cf83af2ad8e9',
   u'kwargs': u'{}',
   u'name': u'task_name',
   u'time_start': 9636801.218162088,
   u'worker_pid': 7931}]}

The time_start attribute dates the task back to 1970 (that's before the creation of Celery, Python, and I don't own a customised DeLorean):

>> from datetime import datetime
>> datetime.fromtimestamp(9636801.218162088)
datetime.datetime(1970, 4, 22, 13, 53, 21, 218162)

Am I misinterpreting the time_task attribute? Is my Celery app misconfigured?

I am using Celery 3.1.4 on Linux with a Django app and a Redis backend.

Tasks are run by a worker that is executed as follows:

./manage.py celery worker --loglevel=INFO --soft-time-limit=600 --logfile=/tmp/w1.log --pidfile=/tmp/w1.pid -n 'w1.%%h'

Answer 1

I found the answer to my own question by digging in the Celery and Kombu code: the time_start attribute of a task is computed by the kombu.five.monotonic function. (Ironically, the kombu code also refers to another StackOverflow question for reference) The timestamp returned by that function refers to a "monotonic" time computed by the clock_gettime system call.

As explained in the clock_gettime documentation, this monotonic time represents the time elapsed "since some unspecified starting point". The purpose of this function is to make sure that time increases monotonically, despite changes of other clock values.

Thus, in order to obtain the real datetime at which the task was started, we just need to compare the time_start attribute to the current value of the monotonic clock:

>> from datetime import datetime
>> from time import time
>> import kombu.five
>> datetime.fromtimestamp(time() - (kombu.five.monotonic() - 9636801.218162088))
datetime.datetime(2013, 11, 20, 9, 55, 56, 193768)

EDIT: the time_start attribute reported by inspection is no longer monotonic : https://github.com/celery/celery/pull/3684 And it only took me four years to write a proper pull request 0:-)