Question

How can I recursively list all all the contents of a bucket using the AWS CLI similar to using find . on Unix.

aws s3 ls s3://MyBucket --recursive complains with unknown option.

http://docs.aws.amazon.com/cli/latest/reference/s3/index.html#directory-and-s3-prefix-operations claims that --recursive is a valid parameter.

Was it helpful?

Solution

aws s3 ls s3://MyBucket --recursive works fine for me.

Try updating your AWS CLI. My version is aws-cli/1.6.2

aws --version

OTHER TIPS

With recent AWS CLI versions, --recursive option is supported.

You can list recursively all the files under a bucket named MyBucket using following command:

aws s3 ls s3://MyBucket/ --recursive

You can list recursively all the files under a folder named MyFolder in the bucket, using following command:

aws s3 ls s3://MyBucket/MyFolder/ --recursive

As @Francisco Cardoso said, the final / is very important. It allows to list the content of the folder instead of the folder itself

For more information, see: https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html

I am not able to interpret the link you referred properly: http://docs.aws.amazon.com/cli/latest/reference/s3/index.html#directory-and-s3-prefix-operations

However, I was able to make --recursive option work with respect to this link: http://docs.aws.amazon.com/cli/latest/reference/s3/index.html#single-local-file-and-s3-object-operations

as per this link, cp, mv and rm supports --recursive option.

The one that you are trying is ls.

I tried using cp and rm with --recursive option and it is working fine.

You can not list recursively all the contents of a bucket via -

aws s3 ls s3://MyBucket

To list object from a folder you need to execute command as -

aws s3 ls s3://MyBucket/MyFolder/

This above command lists object that reside inside folder named MyFolder.

To get an objects list from such a logical hierarchy from Amazon S3, you need specify the full key name for the object in the GET operation.

--recursive Command is performed on allfiles or objects under the specified directory or prefix.

Thanks

Below one line bash script is able to perform:- how to list all S3 buckets with their objects recursively, list bucket name and count objects also.

/usr/bin/sudo /usr/local/bin/aws s3 ls |awk '{print $NF}'| while read l;do echo -e "#######---$l objects---##########\n\n";/usr/bin/sudo /usr/local/bin/aws s3 ls $l|nl;done
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top