gearman or custom mongo

Question

This is not a this or that sort of question trying to find out what I am missing in the difference and understanding. Normally when i want to distributed work and queue things I would write the jobs to be done into a database like mysql or mongodb and then i will have something setup that i can lock a row record and unlock it to avoid two nodes working on the same thing. Now i have heard about gearman recently and trying to find what the differences between that vs a simple database solution. Anyone can tell me what i am missing?

The application I am working on is taking a lot of complex log data and being able to run some analytic on it. to avoid an overload i dont want to process on the spot. So need them queued I just normally write them to the database and wait so now i am trying to get a soild understand on what the differance is and why what i was doing was the "wrong way"

Answer 1

The biggest difference is that gearman is not a database. I know you know this but I thought I would reiterate it. Gearman is a separate entity that must run as a deamon on its own.

Gearman is a distributed work queue and worker already pre-made whereas a database like MySQL or MongoDB has no real built in distributed work queuer and worker built in.

Gearman has a distributed job enrollment and replication system which normal databases do not. And it can also distribute (as I said) workers and jobs evenly across a massive cluster (think of cluster processing on AWS spot instances).

When not using Gearman you would probably have a headache trying to implement all of the things that gearman does in your own code, especially in PHP which is terrible at running in deamon mode.

So with Gearman you get a full community developed and vetted distributed work queue and worker whereas with your own code...well I am not sure what you would get since I am unsure what you input into your work queue and worker.

Edit

The use cases of Gearman over a simple solution would be:

• Distributed video encoder
• Scientific cluster computation of equations
• General Pooling of computational hardware to create a stack to process other tasks

Really Gearman shines in distributed environments. It is like the MongoDB of distributed job queues and workers. It just scales.

That being said if you are running a simple app with a none large scale distributed cluster of workers then Gearman could be overkill. But from my own personal experience of using it Gearman is not that much harder to setup than your own queue.