global distributed lock that can be set to expire Java

Question

I have a use case where I want to have a globally distributed lock. We started out using SELECT .. FOR UPDATE, but that quickly started to have problems as we scaled up the number of servers. Also it didn't account for processes that checked out the lock and then died and failed to return the lock.

We need to be able to set an expiration on the lock (i.e. if the process who checked out the lock does not return it in 2 hours, the lock is automatically returned to the pool). I realize that this introduces the issue where we are ignoring locks, but we are fairly certain that the process has died if not complete in 2 hours. Also the job is idempotent, so if it is done more than once it's not a big deal.

I've looked through a number of distributed locking systems and come across this questions that have been extremely helpful. All of the solutions extend off of Java's java.util.concurrency.locks.Lock, which actually may be the issue I'm coming across because that interface doesn't have the expiration feature I need. We have a similar strategy to mongo-java-distributed-lock where we use MongoDB's findAndModify. We're considering:

• mongo-java-distributed-lock
• Redisson
• hazelcast

as our distributed locking mechanism (all happen to implement java.util.concurrency.locks.Lock).

The biggest problem is that because java.util.concurrency.locks.Lock doesn't have an option for expiring a lock, these don't fit all the goals. This answer probably gets closest with hazelcast, but it is reliant on an entire server failing, not just a thread taking too long. Another option is possibly using a Samaphore with hazelcast as described here. I could have a reaper thread that is then able to cancel the locks of others if they are taking too long. With Mongo and Redis I could take advantage of their ability to expire objects, but that doesn't seem to be part of either of the libraries since they just implement java.util.concurrency.locks.Lock in the end.

So this was just a long winded way of asking, is there a distributed locking mechanism out there that I can have automatically expire after N seconds? Should I be looking at a different mechanism than java.util.concurrency.locks.Lock in this situation altogether?

Answer 1

You may use Redisson based on Redis server. It implements familiar Java data structures including java.util.Lock with distributed and scalable abilities. Including ability to setup lock release timeout. Usage example:

Config config = new Config();
// for single server
config.useSingleServer()
      .setAddress("127.0.0.1:6379");
// or 
// for master/slave servers
config.useSentinelConnection()
      .setMasterName("mymaster")
      .addSentinelAddress("127.0.0.1:26389", "127.0.0.1:26379");

Redisson redisson = Redisson.create(config);

Lock lock = redisson.getLock("anyLock");
try {
   // unlock automatically after 10 seconds of hold
   lock.lock(10, TimeUnit.SECONDS);

} finally {
   lock.unlock();
}

...

redisson.shutdown();

Answer 2

You should consider using zookeeper. And there is an easy to use library for these kind of "distributed" stuff which is built on top of zookeeper : curator framework. I think what you are looking for is shared reentrant lock. You can also check other locks in recipes.

Answer 3

What about this one? http://www.gemstone.com/docs/5.5.0/product/docs/japi/com/gemstone/gemfire/distributed/DistributedLockService.html

Its lock method seems to have what you need:

public abstract boolean lock(Object name, long waitTimeMillis, long leaseTimeMillis)

Attempts to acquire a lock named name. Returns true as soon as the lock is acquired. If the lock is currently held by another thread in this or any other process in the distributed system, or another thread in the system has locked the entire service, this method keeps trying to acquire the lock for up to waitTimeMillis before giving up and returning false. If the lock is acquired, it is held until unlock(Object name) is invoked, or until leaseTimeMillis milliseconds have passed since the lock was granted - whichever comes first.

Answer 4

Actually, as far as I can tell mongo-java-distributed-lock has the ability to expire a lock through the use of DistributedLockOptions.setInactiveLockTimeout()

I haven't tried it yet, but think I will...

EDIT: I have now also tried it, and it works well...

String lockName = "com.yourcompany.yourapplication.somelock";
int lockTimeoutMilliSeconds = 500;

String dbURI = CWConfig.get().getMongoDBConfig().getDbURI();
DistributedLockSvcFactory lockSvcFactory = new DistributedLockSvcFactory(new DistributedLockSvcOptions(dbURI));
DistributedLockSvc lockSvc = lockSvcFactory.getLockSvc();

DistributedLock lock = lockSvc.create(lockName);
lock.getOptions().setInactiveLockTimeout(lockTimeoutMilliSeconds);
try {
    lock.lock();
    // Do work
} finally {
    lock.unlock();
}