flock(): removing locked file without race condition?

Question

I'm using flock() for inter-process named mutexes (i.e. some process can decide to hold a lock on "some_name", which is implemented by locking a file named "some_name" in a temp directory:

lockfile = "/tmp/some_name.lock";
fd = open(lockfile, O_CREAT);
flock(fd, LOCK_EX);

do_something();

unlink(lockfile);
flock(fd, LOCK_UN);

The lock file should be removed at some point, to avoid filling the temp directory with hundreds of files.

However, there is an obvious race condition in this code; example with processes A, B and C:

A opens file
A locks file
B opens file
A unlinks file
A unlocks file
B locks file (B holds a lock on the deleted file)
C opens file (a new file one is created)
C locks file (two processes hold the same named mutex !)

Is there a way to remove the lock file at some point without introducing this race condition ?

Answer 1

Sorry if I reply to a dead question:

After locking the file, open another copy of it, fstat both copies and check the inode number, like this:

lockfile = "/tmp/some_name.lock";

    while(1) {
        fd = open(lockfile, O_CREAT);
        flock(fd, LOCK_EX);

        fstat(fd, &st0);
        stat(lockfile, &st1);
        if(st0.st_ino == st1.st_ino) break;

        close(fd);
    }

    do_something();

    unlink(lockfile);
    flock(fd, LOCK_UN);

This prevents the race condition, because if a program holds a lock on a file that is still on the file system, every other program that has a leftover file will have a wrong inode number.

I actually proved it in the state-machine model, using the following properties:

If P_i has a descriptor locked on the filesystem then no other process is in the critical section.

If P_i is after the stat with the right inode or in the critical section it has the descriptor locked on the filesystem.

Answer 2

1. In Unix it is possible to delete a file while it is opened - the inode will be kept until all processes have ended that have it in their file descriptor list
2. In Unix it is possible to check that a file has been removed from all directories by checking the link count as it becomes zero

So instead of comparing the ino-value of the old/new file paths you can simply check the nlink count on the file that is already open. It assumes that it is just an ephemeral lock file and not a real mutex resource or device.

lockfile = "/tmp/some_name.lock";

for(int attempt; attempt < timeout; ++attempt) {
    int fd = open(lockfile, O_CREAT, 0444);
    int done = flock(fd, LOCK_EX | LOCK_NB);
    if (done != 0) { 
        close(fd);
        sleep(1);     // lock held by another proc
        continue;
    }
    struct stat st0;
    fstat(fd, &st0);
    if(st0.st_nlink == 0) {
       close(fd);     // lockfile deleted, create a new one
       continue;
    }
    do_something();
    unlink(lockfile); // nlink :=0 before releasing the lock
    flock(fd, LOCK_UN);
    close(fd);        // release the ino if no other proc 
    return true;
}
return false;

Answer 3

If you use these files for locking only, and do not actually write to them, then I suggest you treat the existence of the directory entry itself as an indication for a held lock, and avoid using flock altogether.

To do so, you need to construct an operation which creates a directory entry and reports an error if it already existed. On Linux and with most file systems, passing O_EXCL to open will work for this. But some platforms and some file systems (older NFS in particular) do not support this. The man page for open therefore suggests an alternative:

Portable programs that want to perform atomic file locking using a lockfile, and need to avoid reliance on NFS support for O_EXCL, can create a unique file on the same file system (e.g., incorporating hostname and PID), and use link(2) to make a link to the lockfile. If link(2) returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also successful.

So this looks like a locking scheme which is officially documented and therefore indicates a certain level of support and best practice suggestion. But I have seen other approaches as well. bzr for example uses directories instead of symlinks in most places. Quoting from its source code:

A lock is represented on disk by a directory of a particular name, containing an information file. Taking a lock is done by renaming a temporary directory into place. We use temporary directories because for all known transports and filesystems we believe that exactly one attempt to claim the lock will succeed and the others will fail. (Files won't do because some filesystems or transports only have rename-and-overwrite, making it hard to tell who won.)

One downside to the above approaches is that they won't block: a failed locking attempt will result in an error, but not wait till the lock becomes available. You will have to poll for the lock, which might be problematic in the light of lock contention. In that case, you might want to further depart from your filesystem-based approach, and use third party implementations instead. But general questions on how to do ipc mutexes have already been asked, so I suggest you search for [ipc] [mutex] and have a look at the results, this one in particular. By the way, these tags might be useful for your post as well.