Beanstalk and Time Syncronization between computers on LAN

Question

Long story short I've had a lot of issues with my tools hitting my MySQL DB server too hard. So I created a set of tools (PHP and Ruby) that use Beanstalk to communicate what I want done on the DB to a set of always running DB connections. Beanstalk tubes are used for both the incoming jobs and the replies.

However, I've added an ExpireTime to my DB requests' job data. In essence if the DB request job is not executed within a fairly short time window the request is not executed at all. The requesting task will timeout on its own anyway so having a short window of time keeps the DB execution script from executing a bunch of DB requests that will never be used.

This has all worked just fine as long as everything was on one machine. Now I'm expanding it to a second machine and I immediately run into issues where the requests are deemed as having expired.

The current time to expire for a task is 10 seconds - which is already pretty darn long for the simple DB requests I'm attempting to complete. If I increase the 10 seconds to a large enough number then DB requests execute quickly and without issue.

So I suspect this issue boils down to 1 computer thinking it is 12:00:XX and another thinking it is 12:00:YY and the difference between XX and YY is at least 8 or 9 seconds.

I'm using Ubuntu 12.04 desktops and servers in my LAN. I write programs in both Ruby (1.8.7) and PHP (5.3.10-1ubuntu3.4) that use this Beanstalk queue as a sort of DB connection pool as well as a task queue for executing project/site related tasks in a distributed manner. For PHP I use pheanstalk and for Ruby I use the beankstalk-client gem.

A hack I've considered is to create a beanstalk tube where once a second or so the computer that executes the DB requests puts out it's current time. Then the distributed computers that are putting requests into the queue would need to grab that job (and release it - not delete it) for the most current timestamp. This is crude and (I believe) would lead to collisions - i.e. requests waiting on getting the job data for the current time - as I expand the number of computers I add to this.

I'm hoping someone on this forum has a better way of doing this.

Answer 1

The 'correct' solution is to setup an NTPd daemon on the machines (or at least regularly run the program ntpdate -s _servername_) so that the servers clock is correct, and keeps getting corrected when it does get out of alignment.