I don't exactly what caused the issue. I spend quite a bit of time on this but could not find why the job was not running. Eventually I looked at the log and found that the job that was run ahead of it was throwing some kind of exception, so I moved my job from last in the list to first in the list and it started working fine. There were a couple of weird things that happened as well and I was not completely satisfied how it got resolved. Other things that I think of was
- We had a number of environments (dev, staging, staging), it is possible there was some kind of conflict caused by these multiple environments however I think it is unlikely
Anyways the job start running fine after I reorder the job. The interesting thing was, I fixed the ordering in a branch and it got fixed in dev and staging environments.