S3cmd sync returns "killed"

Question

I am trying to sync a few large buckets on amazon S3.

When I run my S3cmd sync --recursive command I get a response saying "killed".

Does anyone know what this may refer to? Is there a limit on the number of files that can be synced in S3?

Thx for your help

Answer 1

After reading around it looks like the program has memory consumption issues. In particular this can cause the OOM killer (out of memory killer) to take down the process and prevent the system from getting bogged down. A quick look at dmesg after the process is killed will generally show if this is the case or not.

With that in mind I would ensure you're on the latest release, which notes memory consumption issues being solved in the release notes.

Answer 2

Old question, but I would like to say that, before you try to add more physical memory or increase vm memory, try just adding more swap.

I did this with 4 servers (ubuntu and centos) with low ram (700MB total, only 15MB available) and it is working fine now.