AWS CLI get download S3 URL for private bucket from AWS CLI

StackOverflow https://stackoverflow.com/questions/21392796

Question

I could upload a file to a private S3 bucket successfully using following command:

aws s3 cp "myfile.txt" "s3://myfolder/myfile.txt" --region=us-east-1 --output=json

I would like to issue a AWS CLI command to return me a temporary URL download for myfile.txt and does anyone know how to?

I googled and look like I have to do some signing to get temporary URL such as: http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html

Was it helpful?

Solution 3

I googled around and decided write this script to help me to generate singed-url for S3.

https://github.com/gdbtek/aws-tools

OTHER TIPS

aws cli now supports presign command. You can run 

$ aws s3 presign s3://test-bucket/test-file.txt
https://test-bucket/test-file.txt?Expires=1499152189&Signature=some-sha

This will generate an url which you can share it with anyone to download that file in 3600 seconds.

You can change time period with --expires-in

$ aws s3 presign s3://test-bucket/test-file.txt --expires-in 600

The generated url will expire in 10 minutes.

You can read more about presign in aws cli docs.

You can use the following URL format:

https://<bucket-name>.s3.amazonaws.com/<object or key name>

or old style:

https://s3.amazonaws.com/<bucket-name>/<object or key name>

To have it accessible you need to allow public access to your object or attach an appropriate bucket policy.

For example the following bucket policy shows public access to bucket zzzyyy object 'yyyeee'

$ aws s3 get-object-acl --bucket zzzyyy --key yyyeee
{
    "Owner": {
        "DisplayName": "owner",
        "ID": "Some hash of owner"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "owner",
                "ID": "Some hash of owner"
            },
            "Permission": "READ"
        },
        {
            "Grantee": {
                "DisplayName": "owner",
                "ID": "Some hash of owner"
            },
            "Permission": "WRITE"
        },
        {
            "Grantee": {
                "DisplayName": "owner",
                "ID": "Some hash of owner"
            },
        "Permission": "READ_ACP"
        },
        {
            "Grantee": {
                "DisplayName": "owner",
                "ID": "Some hash of owner"
            },
            "Permission": "WRITE_ACP"
        },
        {
            "Grantee": {
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"
        },
        {
            "Grantee": {
                "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ_ACP"
        }
    ]
}

You can see examples of bucket policies here:

http://docs.aws.amazon.com/AmazonS3/latest/dev/AccessPolicyLanguage_UseCases_s3_a.html

You can also use the S3 Console as seen here:

S3 Console

The command is wrong; instead of:

$ aws s3 get-object-acl --bucket zzzyyy --key yyyeee

It should be:

$ aws s3api get-object-acl --bucket zzzyyy --key yyyeee

Alternatively you can take a look at https://github.com/minio/mc and give us feedback. We always wanted to have a CLI dropbox interface, of course not as GUI but scriptable and to be automated. So we ended up implementing a convenient command for mc to address two use cases.

  • For sharing URLs to download files (Presigned Get)
  • For sharing curl form to upload files (Presigned POST Policy)

Following examples address both use cases.

$ mc share
NAME:
  mc share - Download and upload documents.

USAGE:
  mc share command [arguments...]

COMMANDS:
  download  Generate URL to download documents.
  upload    Generate ‘curl’ command to upload files.
  list      List the shared URLs

These commands can be used in regular sense or also from with in a script. For scripting purposes mc provides --json output for all the commands.

Example for mc share download

$ mc share download https://s3.amazonaws.com/ferenginar/sur_2013-08-05T05_48_49-07_00.mp3
https://s3.amazonaws.com/ferenginar/sur_2013-08-05T05_48_49-07_00.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJVA5BMMU2RHO6IOQ%2F20151109%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151109T060251Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3c69928bee38b7b465547f9fa97146ff606480bbead8042c0351840c578a2ceb

$ mc --json share download https://s3.amazonaws.com/ferenginar/sur_2013-08-05T05_48_49-07_00.mp3 | jq .

{
  "expiry": {
    "days": 7
  },
  "downloadUrl": "https://s3.amazonaws.com/ferenginar/sur_2013-08-05T05_48_49-07_00.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJVA5BMMU2RHO6IOQ%2F20151109%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20151109T060618Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=48f7a521396ac8e36443cc86ce16c2619f9fa81371aad9a309375c933ea48d99",
  "keyName": "https://s3.amazonaws.com/ferenginar/sur_2013-08-05T05_48_49-07_00.mp3"
}

Example for mc share upload

$ mc share upload --expire=2h --content-type=image/png https://s3.amazonaws.com/backup/2007-Mar-2/... 
curl https://s3.amazonaws.com/backup -F x-amz-credential=AKIAJVA5BMMU2RHO6IOQ/20151109/us-east-1/s3/aws4_request -F x-amz-date=20151109T061202Z -F x-amz-signature=61c234dd17fc9e5888417ac4546b37d61755e551674a9617f33f5232ec53c8b6 -F Content-Type=image/png -F bucket=backup -F policy=eyJleHBpcmF0aW9uIjoiMjAxNS0xMS0wOVQwODoxMjowMi40MDZaIiwiY29uZGl0aW9ucyI6W1siZXEiLCIkQ29udGVudC1UeXBlIiwiaW1hZ2UvcG5nIl0sWyJlcSIsIiRidWNrZXQiLCJiYWNrdXAiXSxbInN0YXJ0cy13aXRoIiwiJGtleSIsIjIwMDctTWFyLTIvIl0sWyJlcSIsIiR4LWFtei1kYXRlIiwiMjAxNTExMDlUMDYxMjAyWiJdLFsiZXEiLCIkeC1hbXotYWxnb3JpdGhtIiwiQVdTNC1ITUFDLVNIQTI1NiJdLFsiZXEiLCIkeC1hbXotY3JlZGVudGlhbCIsIkFLSUFKVkE1Qk1NVTJSSE82SU9RLzIwMTUxMTA5L3VzLWVhc3QtMS9zMy9hd3M0X3JlcXVlc3QiXV19 -F x-amz-algorithm=AWS4-HMAC-SHA256 -F key=2007-Mar-2/<FILE> -F file=@<FILE>

$ mc --json share upload --expire=2h --content-type=image/png https://s3.amazonaws.com/backup/2007-Mar-2/... | jq .
{
  "expiry": {
    "hours": 2
  },
  "uploadCommand": "curl https://s3.amazonaws.com/backup -F x-amz-signature=4b349a15e76ab403424ec6b1e289336f67c02c427ccc98522d6f654eee5b7de6 -F Content-Type=image/png -F bucket=backup -F policy=eyJleHBpcmF0aW9uIjoiMjAxNS0xMS0wOVQwODoxMzoyMS4zODFaIiwiY29uZGl0aW9ucyI6W1siZXEiLCIkQ29udGVudC1UeXBlIiwiaW1hZ2UvcG5nIl0sWyJlcSIsIiRidWNrZXQiLCJiYWNrdXAiXSxbInN0YXJ0cy13aXRoIiwiJGtleSIsIjIwMDctTWFyLTIvIl0sWyJlcSIsIiR4LWFtei1kYXRlIiwiMjAxNTExMDlUMDYxMzIxWiJdLFsiZXEiLCIkeC1hbXotYWxnb3JpdGhtIiwiQVdTNC1ITUFDLVNIQTI1NiJdLFsiZXEiLCIkeC1hbXotY3JlZGVudGlhbCIsIkFLSUFKVkE1Qk1NVTJSSE82SU9RLzIwMTUxMTA5L3VzLWVhc3QtMS9zMy9hd3M0X3JlcXVlc3QiXV19 -F x-amz-algorithm=AWS4-HMAC-SHA256 -F x-amz-credential=AKIAJVA5BMMU2RHO6IOQ/20151109/us-east-1/s3/aws4_request -F x-amz-date=20151109T061321Z -F key=2007-Mar-2/<FILE> -F file=@<FILE> ",
  "keyName": "https://s3.amazonaws.com/backup/2007-Mar-2/..."
}

Looking forward on some feedback. Thanks

Know this is 8 months late, but I just found out you can use Eclipse with the AWS Toolkit for Eclipse (http://aws.amazon.com/eclipse/) to generate pre-signed URLs with a right-click on the target file, even lets you set the expy date....It's pretty slick.

I don't think there is any pre built cli command \sub command to get private bucket url, for pre-sign you can refer presign . In case if you want a public url in private bucket, that is object read is public not bucket. For public aws url, you can use something - I have written this in powershell, you can put this $profile

Function getAwsUrl{
[CmdletBinding(DefaultParameterSetName="S3Path")]
    param(
    [parameter (ParameterSetName="S3Path",mandatory=$true)]$s3path,
    [parameter (ParameterSetName="BucketKey")]$bucket,
    [parameter (ParameterSetName="BucketKey",mandatory=$true)]$key 
    )
if($s3path)
{
$splits = $s3path.Split('//')
$bucket =$splits[2]
$key = $splits[3]
}
else 
{
#user input bucket and key
$bucket =$bucket
$key = $key
}

#using standard aws region "us-west-2"
Write-Output "https://$bucket.s3.us-west-2.amazonaws.com/$key"
}

Example usage: Make sure you pass --acl as public-read

aws s3 cp C:\temp\dumy.txt s3://fpd-uploads/ --acl public-read
getAwsUrl -s3path s3://fpd-uploads/dumy.txt

get-help getAwsUrl

NAME
    getAwsUrl

SYNTAX
    getAwsUrl -s3path <Object>  [<CommonParameters>]

    getAwsUrl -key <Object> [-bucket <Object>]  [<CommonParameters>]


ALIASES
    None


REMARKS
    None
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top