gsutil cp: concurrent execution leads to local file corruption

Question

I have a Perl script which calls 'gsutil cp' to copy a selected from from GCS to a local folder:

$cmd = "[bin-path]/gsutil cp -n gs://[gcs-file-path] [local-folder]";
$output = `$cmd 2>&1`;

The script is called via HTTP and hence can be initiated multiple times (e.g. by double-clicking on a link). When this happens, the local file can end up being exactly double the correct size, and hence obviously corrupt. Three things appear odd:

1. gsutil seems not to be locking the local file while it is writing to it, allowing another thread (in this case another instance of gsutil) to write to the same file.

2. The '-n' seems to have no effect. I would have expected it to prevent the second instance of gsutil from attempting the copy action.

3. The MD5 signature check is failing: normally gsutil deletes the target file if there is a signature mismatch, but this is clearly not always happening.

The files in question are larger than 2MB (typically around 5MB) so there may be some interaction with the automated resume feature. The Perl script only calls gsutil if the local file does not already exist, but this doesn't catch a double-click (because of the time lag for the GCS transfer authentication).

gsutil version: 3.42 on FreeBSD 8.2

Anyone experiencing a similar problem? Anyone with any insights?

Edward Leigh

Answer 1

1) You're right, I don't see a lock in the source.

2) This can be caused by a race condition - Process 1 checks, sees the file is not there. Process 2 checks, sees the file is not there. Process 1 begins upload. Process 2 begins upload. The docs say this is a HEAD operation before the actual upload process -- that's not atomic with the actual upload.

3) No input on this.

You can fix the issue by having your script maintain an atomic lock of some sort on the file prior to initiating the transfer - i.e. your check would be something along the lines of:

use Lock::File qw(lockfile);

if (my $lock = lockfile("$localfile.lock", { blocking => 0 } )) {
     ... perform transfer ...
     undef $lock;
}
else {
    die "Unable to retrieve $localfile, file is locked";
}

Answer 2

1) gsutil doesn't currently do file locking.

2) -n does not protect against other instances of gsutil run concurrently with an overlapping destination.

3) Hash digests are calculated on the bytes as they are being downloaded as a performance optimization. This avoids a long-running computation once the download completes. If the hash validation succeeds, you're guaranteed that the bytes were written successfully at one point. But if something (even another instance of gsutil) modifies the contents in-place while the process is running, the digesters will not detect this.

Answer 3

Thanks to Oesor and Travis for answering all points between them. As an addendum to Oesor's suggested solution, I offer this alternative for systems lacking Lock::File:

use Fcntl ':flock'; # import LOCK_* constants

# if lock file exists ...
if (-e($lockFile))
{
  # abort if lock file still locked (or sleep and re-check)
  abort() if !unlink($lockFile);
  # otherwise delete local file and download again
  unlink($filePath);
}

# if file has not been downloaded already ...
if (!-e($filePath))
{
  $cmd = "[bin-path]/gsutil cp -n gs://[gcs-file-path] [local-dir]";

  abort() if !open(LOCKFILE, ">$lockFile");
  flock(LOCKFILE, LOCK_EX);
  my $output = `$cmd 2>&1`;
  flock(LOCKFILE, LOCK_UN);
  unlink($lockFile);
}