scp a range of files

Question

For example, in my remote place, there are lots of dump files named "file_1" to "file_50500000", and I only need files from file_1000000 to file_50500000 with an increment as 100000.

#!/bin/bash
files=""
for i in `seq -f '%9.0f' 1000000 100000 50500000`; do
files="$files file_$i.dump"
done
scp user@abc.abc.com:/user/$files ./

The error is "No such file or directory". Thank you for any suggestions in advance.

Answer 1

Your for-loop adds to $files like this:

files="$files file_$i.dump"

# $files will look like this when the loop is done:
" file_1000000.dump file_1100000.dump file_1200000.dump ..."

Now look what happens when you use that with scp:

scp user@abc.abc.com:/user/ file_1000000.dump file_1100000.dump file_1200000.dump ./

That will clearly not work.

If your files really are in subdir /user (sure you don't mean ~user?), you can use {} to group them. Add them to $files in your loop like this:

files="$files,file_$i.dump"
# When finished, it will look like this (yes leading ,)
",file_1000000.dump,file_1100000.dump,file_12000000.dump,..."

Now call scp using bash variable substitution to remove the leading ,

scp user@abc.abc.com:/user/{${files#,}} ./

If this feels too complicated and you don't mind the extra transfer-time, maybe just scp the files one by one in the loop, as suggested in an other answer...

Answer 2

You have to provide full paths to scp and you may have a quoting issue, try

#!/bin/bash
files="/user/file_1000000.dump"
for i in `seq -f '%9.0f' 1100000 100000 50500000`; do
  files="$files /user/file_$i.dump"
done
scp "user@abc.abc.com:/$files" ./

e.g. Output using some empty touched dump files on another computer

> ./script
user password:
file_1000000.dump                             100%    0     0.0KB/s   00:00    
file_1100000.dump                             100%    0     0.0KB/s   00:00    
file_1200000.dump                             100%    0     0.0KB/s   00:00    
file_1300000.dump                             100%    0     0.0KB/s   00:00    
file_1400000.dump                             100%    0     0.0KB/s   00:00    
file_1500000.dump                             100%    0     0.0KB/s   00:00  
...

Answer 3

"/user" is missing in path of remote files. It is only in first file path and actually I am not sure that is possible to specify files for scp this way. I prefer:

for i in `seq -f '%9.0f' 1000000 100000 50500000`; do
    scp user@abc.abc.com:/user/file_$i.dump .
done

in case that files are small there is another way to avoid overhead with establishing of connection

ssh user@abc.abc.com "cd /user/; tar cf - $(seq -f '%9.0f' 1000000 100000 50500000)" |tar xf -

it create tar archive and write it to standard output that is redirected to the local machine and where another tar write files from the stdout to files. Option for compression "z" can be used for both tar if it is supported or gzip and gunzip can be added to the pipe.