Synchronize shell script execution

StackOverflow https://stackoverflow.com/questions/1474052

Question

A modified version of a shell script converts an audio file from FLAC to MP3 format. The computer has a quad-core CPU. The script is run using:

./flac2mp3.sh $(find flac -type f)

This converts the FLAC files in the flac directory (no spaces in file names) to MP3 files in the mp3 directory (at the same level as flac). If the destination MP3 file already exists, the script skips the file.

The problem is that sometimes two instances of the script check for the existence of the same MP3 file at nearly the same time, resulting in mangled MP3 files.

How would you run the script multiple times (i.e., once per core), without having to specify a different file set on each command-line, and without overwriting work?

Update - Minimal Race Condition

The script uses the following locking mechanism:

  # Convert FLAC to MP3 using tags from flac file.
  #
  if [ ! -e $FLAC.lock ]; then
    touch $FLAC.lock
    flac -dc "$FLAC" | lame${lame_opts} \
      --tt "$TITLE" \
      --tn "$TRACKNUMBER" \
      --tg "$GENRE" \
      --ty "$DATE" \
      --ta "$ARTIST" \
      --tl "$ALBUM" \
      --add-id3v2 \
      - "$MP3"
    rm $FLAC.lock
  fi;

However, this still leaves a race condition.

Was it helpful?

Solution

The "lockfile" command provides what you're trying to do for shell scripts without the race condition. The command was written by the procmail folks specifically for this sort of purpose and is available on most BSD/Linux systems (as procmail is available for most environments).

Your test becomes something like this:

lockfile -r 3 $FLAC.lock
if test $? -eq 0 ; then
  flac -dc "$FLAC" | lame${lame_opts} \
    --tt "$TITLE" \
    --tn "$TRACKNUMBER" \
    --tg "$GENRE" \
    --ty "$DATE" \
    --ta "$ARTIST" \
    --tl "$ALBUM" \
    --add-id3v2 \
    - "$MP3"
fi
rm -f $FLAC.lock

Alternatively, you could make lockfile keep retrying indefinitely so you don't need to test the return code, and instead can test for the output file for determining whether to run flac.

OTHER TIPS

If you don't have lockfile and cannot install it (in any of its versions - there are several implementations) a robust and portable atomic mutex is mkdir.

If the directory you attempt to create already exists, mkdir will fail, so you can check for that; when creation succeeds, you have a guarantee that no other cooperating process is in the critical section at the same time as your code.

if mkdir "$FLAC.lockdir"; then
    # you now have the exclusive lock
    : critical section
    : code goes here
    rmdir "$FLAC.lockdir"
else
    : nothing? to skip this file
    # or maybe sleep 1 and loop back and try again
fi

For completeness, maybe also look for flock if you are on a set of platforms where that is reliably made available and need a performant alternative to lockfile.

You could implement locking of FLAC files that it's working on. Something like:

if (not flac locked)
  lock flac
  do work
else
  continue to next flac

Send output to a temporary file with a unique name, then rename the file to the desired name.

flac -dc "$FLAC" | lame${lame_opts} \
      --tt "$TITLE" \
      --tn "$TRACKNUMBER" \
      --tg "$GENRE" \
      --ty "$DATE" \
      --ta "$ARTIST" \
      --tl "$ALBUM" \
      --add-id3v2 \
      - "$MP3.$$"
mv "$MP3.$$" "$MP3"

If a race condition leaks through your file locking system every once in a while, the final output will still be the result of one process.

To lock a file process you can create a file with the same name with a .lock extension.

Before starting the encoding check the existence of the .lock file, and optionally make sure the date of the lockfile isn't too old (in case the process dies). If it does not exist, create it before the encoding starts, and remove it after the encoding is complete.

You can also flock the file, but this only really works in c where you are calling flock() and writing to the file then closing and unlocking. For a shell script, you probably are calling another utility to do the writing of the file.

How about writing a Makefile?

ALL_FLAC=$(wildcard *.flac)
ALL_MP3=$(patsubst %.flac, %.mp3, $(ALL_FLAC)
all: $(ALL_MP3)
%.mp3: %.flac
        $(FLAC) ...

Then do

$ make -j4 all

In bash it's possible to set noclobber option to avoid file overwriting.

help set | egrep 'noclobber|-C'

Use a tool like FLOM (Free LOck Manager) and simply serialize your command as below:

flom -- flac ....
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top