s3cmd list of contents - only filenames - perl one liner?

Question

Currently I'm using s3cmd ls s3://location/ > file.txt to get a list of contents of my s3 bucket and save on a txt. However the above returns dates, filesizes paths and filenames.

for example:

2011-10-18 08:52      6148   s3://location//picture_1.jpg

I only need the filenames of the s3 bucket - so on the above example I only need picture_1.jpg.
Any suggestions?

Could this be done with a Perl one liner maybe after the initial export?

Answer 1

File::Listing does not support this format because the designers of this listing format were stupid enough to not simply reuse an existing one. Let's parse it manually instead.

use URI;
my @ls = (
    "2011-10-18 08:52 6148 s3://location//picture_1.jpg\n",
    "2011-10-18 08:52 6148 s3://location//picture_2.jpg\n",
    "2011-10-18 08:52 6148 s3://location//picture_3.jpg\n",
);

for my $line (@ls) {
    chomp $line;
    my $basename = (URI->new((split q( ), $line)[-1])->path_segments)[-1];
}

__END__
picture_1.jpg
picture_2.jpg
picture_3.jpg

---

As oneliner:

perl -mURI -lne 'print ((URI->new((split q( ), $line)[-1])->path_segments)[-1])' < input

Answer 2

Use awk:

s3cmd ls s3://location/ | awk '{ print $4 }' > file.txt

If you have filenames with spaces, try:

s3cmd ls s3://location/ | awk '{ s = ""; for (i = 4; i <= NF; i++) s = s $i " "; print s }' > file.txt

Answer 3

I am sure a specific module is the safer option, but if the data is reliable, you can get away with a one-liner:

Assuming the input is:

2011-10-18 08:52 6148 s3://location//picture_1.jpg
2011-10-18 08:52 6148 s3://location//picture_2.jpg
2011-10-18 08:52 6148 s3://location//picture_3.jpg
...

The one-liner:

perl -lnwe 'print for m#(?<=//)([^/]+)$#'

• -l chomps the input, and adds newline to end of print statements
• -n adds a while(<>) loop around the script
• (?<=//) lookbehind assertion finds a double slash
• ...followed by non-slashes to the end of the line
• The for loop assures us that non-matches are not printed.

The benefit of the -n option is that this one-liner may be used in a pipe, or on a file.

command | perl -lnwe '...'
perl -lnwe '...' filename