Amazon S3 .NET SDK can upload only by 2 streams

Question

I've been trying to boost upload speed to S3 server from my c# app for some time but the maximum I got is 2 simultaneous streams each up to 100 KB/s. Which if frustrating, considering that in a browser you can upload up to 1.0 MB/s. Here's my upload code, which uses UploadPartRequest, invoked in separate thread. Nevetherless, S3 allow only 2 threads to upload at the same time:

List<UploadPartResponse> uploadResponses = new List<UploadPartResponse>();
        List<Thread> uploadThreads = new List<Thread>();

        InitiateMultipartUploadRequest initiateRequest = new InitiateMultipartUploadRequest
        {
            BucketName = BucketName,
            Key = result
        };

        InitiateMultipartUploadResponse initResponse = _S3client.InitiateMultipartUpload(initiateRequest);
        _curContentLength = new FileInfo(pathToResult + result).Length;
        long partSize = 5 * (long)Math.Pow(2, 20); // 5 MB

          long filePosition = 0;
          for (int i = 1; filePosition < _curContentLength; i++)
          {
              if ( filePosition + partSize > _curContentLength )
              {
                  partSize = _curContentLength - filePosition;
              }
              UploadPartRequest uploadRequest = new UploadPartRequest
                  {
                      BucketName = BucketName,
                      UploadId = initResponse.UploadId,
                      PartNumber = i,
                      PartSize = partSize,
                      FilePosition = filePosition,
                      Key = result,
                FilePath = (pathToResult + result)
                  };

                  uploadRequest.StreamTransferProgress += uploadFileProgressCallback;

                  // Upload part and add response to our list.

                  Thread t = new Thread(() => uploadResponses.Add(_S3client.UploadPart(uploadRequest)));
                  t.Start();
                  uploadThreads.Add(t);
                  filePosition += partSize;
              }

              foreach (var uploadThread in uploadThreads)
              {
                  uploadThread.Join();
              }

              // Step 3: complete.
              CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest
                  {
                      BucketName = BucketName,
                      UploadId = initResponse.UploadId,
                      Key = result
                      //,PartETags = new List<PartETag>(uploadResponses)

                  };
              completeRequest.AddPartETags(uploadResponses);
              _S3client.CompleteMultipartUpload(completeRequest);
              _systemEnable = true;
              WriteLineToLog("Успешно залил ролик на S3!", intId);
              break;
          }

is something wrong with the code or it is impossible to upload to s3 with c# with speed larger than 200 KB/s? Thnks!

Answer 1

I think this issue not from Amazon S3 it's just because "HTTP Two-Connection Limit".

You can change it by adding this configuration to your app.config file:

<configuration>  
  <system.net>
    <connectionManagement>
       <add address="*" maxconnection="20" />
    </connectionManagement>
  </system.net>
</configuration>