I found out the issue was an error causing the first worker to do what it was supposed to do but then crashing before removing the job.
So the next worker would pick it up and it would fail jumping to my catch block and bypassing the php error and then delete the job.