Google Cloud Storage Multipart Upload, with Last Chunk Failing (503)

Question

So I am trying to upload a file to Google Cloud storage using multipart uploads, by breaking the file into chunks beforehand and sending it via http put requests.

The problem is that when I upload the very last chunk of the file, I get a 503 error, rather than a 200 complete message.

For simplicity's sake, I am using a jpeg image from drive, converted to a blob and then to bytes:

var data = DriveApp.getFileById(DriveImageFileId).getBlob().getBytes();

Following the documentation, I first request a repeatable upload URI.

  var accessToken = OAuth2.0_Access_Token;
  var header = {'X-Upload-Content-Type' : 'image/jpeg'}
  var body = {"name": "filename.jpeg"}  
  var options = {
    'method':'post',
    'contentType':'application/json',
    'oAuthUseToken':'never',  // Disables OAuth1.0 config 
    'muteHttpExceptions':true,
    'headers':header,
    'contentLength': 582381,
    'payload':JSON.stringify(body)
  };

  var response = UrlFetchApp.fetch('https://www.googleapis.com/upload/storage/v1beta2/b/myCloudStoreBucket/o?uploadType=resumable&access_token='+accessToken,options).getHeaders();
  var uploadUrl = response.Location 

This works fine.

I begin parsing the file bytes into 256x1024 byte chunks, and use the upload URL to upload using PUT via http. That isn't a problem, because every time I upload a chunk, I get a 308 response with the expected number of bytes uploaded. I can do this multiple times, in factors of 256x1024, uploading an entire file right up until the very last chunk.

Here is the UrlFetchApp Options and header on the last chunk, for example:

 var options = {
 "method": "put",
 "contentType": "image/jpeg",
 "contentLength": 58093,
 "oAuthUseToken": "never",
 "muteHttpExceptions": true,
 "headers": {
  "Content-Range": "bytes 524287-582380/582381"
 },
 "payload" : <dataBytes>
 }
 var response = UrlFetchApp.fetch(resumeUrl,options);

Which returns the following:

 Response Type: 503
 Response Header:
 {
 "Date": "Wed, 16 Apr 2014 06:08:43 GMT",
 "Content-Length": "0",
 "Alternate-Protocol": "443:quic",
 "Content-Type": "text/html; charset=UTF-8",
 "Server": "HTTP Upload Server Built on Apr 7 2014 11:59:06 (1396897146)"
 }
No Response body or error message.

The response should be 200 File upload complete. What am I doing wrong?

Answer 1

Answering Own Question because this is being resolved:

So there are two parts that are wrong here.

1. The first is that the last file chunk I was sending last was one byte short. I had been using the byte range that I was sending in the header to also slice the byte array. The very last byte was being missed because of this.

   By adjusting the script so that for the very last file chunk, it grabs the last trailing byte, I was able to get a 200 HTTP response, and the file was successfully uploaded.

2. The second part wrong is that the API was returning blank 503, which are supposed to Service Unavailable errors. According to the Cloud Storage documentation, you should try exponentially backing off service in the case of 503. The API should have been throwing 4xx File Upload errors instead, which is why it took me so long to track down what was going wrong here with the upload.

I reported the 503 bug to Google Cloud Storage team and they'll have a fix out in the next couple of releases for their API.