Job Manager Message Queue over HTTP

Question

I have a worker that can currently only accept jobs/tasks over HTTP. That is, instead of running a daemon listening to TCP ports and just getting raw messages, it only listens to HTTP messages. (I know HTTP is just an additional layer over TCP). So jobs have to be constructed and wrapped around HTTP messages.

I want to use a job manager to queue tasks and send these tasks over HTTP to a pool of the workers as described above.

Is there any job managers that relays tasks over HTTP? I don't mean accepting tasks over HTTP, that doesn't matter, but they must be able to send tasks to workers over HTTP.

There are other functionalities a job manager possesses, such as fault tolerance. And even though the HTTP connection is not persistent, is it possible to replicate all the TCP signals a worker will return to the job manager over HTTP?

One solution I was thinking of having a proxy in between that translates the TCP messages into HTTP messages. But this seemed difficult to do.

Answer 1

I believe that better architecture would be mature job queue + wrapper for your worker's API.

1. You choose job scheduler/queue with your requirements (Celery or whatever you like)
2. Write a wrapper script which is able to submit jobs to your worker, report worker's status, etc.