S3 uploader throwing exception while trying to upload huge files

Question

I was trying to upload a huge file (8GB) using Amazon sdk for MAC (though they dnt have an sdk for MAC, I have modified the IOS sdk source code by removing the UIKit references and recompiled for MAC). It's actually working for files, maybe upto 800MB ~ 1GB. But whenever I tried to upload a 8GB file, I am getting an exception. The exception says that "Internet connection is lost". Weird, I am having good internet connection (2Mbps upload speed :-/). I am using multipart upload api and uploading chunks of 5MB. When I get an exception, I am not giving up but I will retry the same chunk for at least 5 times and then give up. In between this retry I am sleeping for a few seconds. But the most amazing part is, once a chunk upload face an exception, it won't really work in retry. I don't understand it. Can any one help me out????

Thanks in advance!

Below am adding the code too!

        fileHandle = [NSFileHandle fileHandleForReadingAtPath:filePath];

        S3InitiateMultipartUploadRequest* initReq = [[S3InitiateMultipartUploadRequest alloc] initWithKey:requestDetails.md5 inBucket:requestDetails.bucket];
        initReq.contentType = requestDetails.contentType;

        S3MultipartUpload* upload = [amazonS3Client_ initiateMultipartUpload:initReq].multipartUpload;
        S3CompleteMultipartUploadRequest* compReq = [[S3CompleteMultipartUploadRequest alloc] initWithMultipartUpload:upload];


        compReq.contentType = requestDetails.contentType;

        NSData* data = [fileHandle readDataOfLength:PART_SIZE];
        int part = 0;
        ...............................
        while([data length] > 0)
        {
                @try {
                    S3UploadPartRequest* upReq = [[S3UploadPartRequest alloc] initWithMultipartUpload:upload];
                    upReq.partNumber = part + 1;
                    upReq.contentLength = data.length;
                    upReq.data = data;
                    upReq.contentType = requestDetails.contentType;

                    response = [amazonS3Client_ uploadPart:upReq];
                    [compReq addPartWithPartNumber:(part + 1) withETag:response.etag];

                    offset += [data length];

                    [fileHandle seekToFileOffset:offset];
                    data = [fileHandle readDataOfLength:PART_SIZE];
                    part++;
                    ........................
                }
                @catch (AmazonClientException *exception) {
                    .............................

                    if([exception.message rangeOfString:@"expire"].location != NSNotFound)
                    {
                        // renew credentials code 
                        return [self multipartUpload:requestDetails withFilePath:filePath];
                    }

                }
            }
      [amazonS3Client_ completeMultipartUpload:compReq];

Answer 1

I am one of the maintainers of the AWS SDK for iOS. While we haven't tested with files of that size I can't think of anything in the SDK that would be causing this. Some things you might want to try and/or include in your question:

1. The latest versions of the AWS SDK for iOS include an option to disable exceptions from the SDK itself. This might show whether the source of exception is in the SDK or something else.

2. You say you are retrying on exception, but are you re-using the request or creating a new one? Including your code may help others offer suggestions.

3. When you get into this state, can you execute any other operations? Are you sure that you haven't caused your network connection to reset in some way?

Follow up

I've spent quite a bit of time trying to reproduce the issue with the latest version of the AWS SDK for iOS over the past few days and haven't been successful. I was able to successfully upload a 10GB file to Amazon S3 using multipart upload.

I wanted to summarize some findings and give some recommendations for next steps:

1. You should really update the SDK. There have been a number of updates for the AWS SDK for iOS over the 6 months since you said you downloaded, some in relation to connection handling, that may help with this problem.

2. Your test for internet connectivity may not be testing the right thing. You did not include the code for this, so it's unclear, but if all you're testing is that the local devices have link and can get to their gateway, that is not necessarily a valid indicator that there isn't a problem with your connection to Amazon S3. The fact that you were getting timeouts (more on this later) is an indication that your computer was having problems connecting to Amazon S3. There are a number of reasons this could happen including faulty network hardware or explicit traffic shaping by your ISP.

3. There's an inconsistency in the SDK with regards to timeout handling. For all clients besides S3, the timeout is applied to the NSURLConnection object as well as processing the actual request. For S3, it is only applied to processing of the request meaning that the NSURLConnection timeoutInverval property is unmodified. The NSURLConnection timeoutInterval property is a timeout to establish the connection. Once data is being sent or received, even if it is just a trickle, NSURLConnection will not timeout. This means that if you are receiving timeout exceptions, your computer was unable to establish a connection to Amazon S3 for the default timeoutInterval (60s). We'll review and patch the SDK as appropriate to address this inconsistency.

4. As I stated in my original answer, it would be good to confirm if you can make another request to Amazon S3 (like a list buckets/objects) as well as get to another external resource like google.com when you get into this failure state. This would at least empower you to go your ISP and ask better questions about what is going on.

Hope this information helps