Question

I have files like this in S3:

1-2013-08-22-22-something
2-2013-08-22-22-something
etc

without srcPattern I can get all of the files from the bucket easily but I want to get a specific prefix, for example all of the 1's. I've tried using srcPattern but for some reason it's not picking up any of the files.

My current command is:

elastic-mapreduce --jobflow $JOBFLOW --jar /home/hadoop/lib/emr-s3distcp-1.0.jar \
--args '--src,s3n://some-bucket/,--dest,hdfs:///hdfs-input,--srcPattern,[0-9]-.*' \
--step-name "copying over s3 files" 
Was it helpful?

Solution

Turns out you need .* in front of the regex

for example I needed

.*[0-9]-.*

I'm guessing because the source pattern also includes the bucket name?

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top