AWS CLI copy "Max retries exceeded...", error 10054: "connection forcibly closed by remote host"

Question

This happens with sync or cp, copying about 1000 files from our local server to an S3 bucket. The source is a mapped network drive from a Windows 8 machine to a Windows Server 2008 R2 VM running in Hyper-V.

This is the command:

aws s3 cp X:\folder s3://my-bucket/folder --recursive

There doesn't appear to be any pattern to the files it fails on. Some are larger (up to 30MB), others are only a few kb. This is the error message:

HTTPSConnectionPool(host='XXX.s3.amazonaws.com', port=443): Max retries exceeded with url: ... (Caused by <class 'socket.error'>: [Errno 10054] An existing connection was forcibly closed by the remote host)

Answer 1

The problem is the mapped network drive. All files upload successfully when copying from a folder on a local disk instead of the network location.

Answer 2

I had the same problem. Notebook with Ubuntu, command aws s3 cp or aws s3 sync were failing as described.

My file was 104285 bytes large, so nothing "too big".

Copying the file over BucketExplorer was working without problems.

On botocore are reported issues, which show similar problems for large files, but no report for small ones.

Then I updated boto library from 2.9.5 to the latest 2.23.0, but the problem has persisted.

Later I was testing my problem file, making it smaller and smaller, until it was published. Finally I have found, the problem is gone even for the original file.

It is hard to say, how and if the problem is resolved.

One can think of a temporary problem on AWS side, but as I was able to publish the file over BucketExplorer on the first try, I guess it was not the case.

Checking AWS Health Dashboard: http://status.aws.amazon.com/ I did not find any problems reported, but you may never rely on finding there all small issues.

Anyway, I would propose updating boto to the latest version as it seems the most likely cause:

 $ pip install --upgrade boto

My AWS CLI was alrady updated before I run into these problems and has version 0.31

My final versions are:

$ pip freeze|grep boto
boto==2.23.0
boto-utils==0.3
botocore==0.31.0