scp with special characters programmatically

Question

I have been searching for this for a while, and can't find a satisfactory answer.

I have a perl script that needs to copy a file from one host to another, essentially

sub copy_file{
    my($from_server, $from_path, $to_server, $to_path, $filename) = @_;

    my $from_location = "$from_server:\"\\\"${from_path}${filename}\\\"\"";
    my $to_location = $to_path . $filename;
    $to_location =~ s/\s/\\\\ /g;
    $to_location = "${to_server}:\"\\\"${to_location}\\\"\"";

    return system("scp -p $from_location $to_location >/dev/null 2>&1"");
}

The problem is, some of my filenames look like this:

BLAH;BLAH;BLAH.TXT
Some really nicely named file( With spaces, prentices, &, etc...).xlx

I am already handling whitespaces, and the code for that is quite ugly since on each side, the files could be local or remote, and the escaping is different for the from and to part of the scp call.

what I am really looking for is either to somehow to escape all possible special characters or somehow bypass the shell expansion entirely by using POSIX system calls. I am ok with writing a XS Module if need be.

I have the correct keys set up in the .ssh directory Also I am not honestly sure which special characters do and don't cause problems. I would like to support all legal filename characters.

Answer 1

Say you want to copy file foo(s) using scp.

As shown below, scp treats the source and target as shell literals, so you pass the following arguments to scp:

• scp
• -p
• --
• host1.com:foo\(s\) or host1.com:'foo(s)'
• host2.com:foo\(s\) or host2.com:'foo(s)'

You can do that using the multi-argument syntax of system plus an escaping function.

use String::ShellQuote qw( shell_quote );

my $source = $from_server . ":" . shell_quote("$from_path/$filename");
my $target = $to_server   . ":" . shell_quote("$to_path/$filename");

system('scp', '-p', '--', $source, $target);

If you really wanted to build a shell command, use shell_quote as usual.

my $cmd = shell_quote('scp', '-p', '--', $source, $target);

---

$ ssh ikegami@host.com 'mkdir foo ; touch foo/a foo/b foo/"*" ; ls -1 foo'
*
a
b

$ mkdir foo ; ls -1 foo

$ scp 'ikegami@host.com:foo/*' foo
*              100%    0     0.0KB/s   00:00
a              100%    0     0.0KB/s   00:00
b              100%    0     0.0KB/s   00:00

$ ls -1 foo
*
a
b

$ rm foo/* ; ls -1 foo

$ scp 'ikegami@host.com:foo/\*' foo
*              100%    0     0.0KB/s   00:00

$ ls -1 foo
*

Answer 2

There are three ways to handle this:

1. Use the multi-argument form of system which will completely avoid the shell:

   system 'scp', '-p', $from_location, $to_location;
   

   Disadvantage: you can't use shell features like redirection.

2. Use String::ShellQuote to escape the characters.

   $from_location = shell_quote $from_location;
   $to_location   = shell_quote $to_location;
   

   Disadvantage: certain strings can exist which can't be quoted safely. Furthermore, this solution is not portable as it assumes Bourne shell syntax.

3. Use IPC::Run which essentially is a supercharged system command that allows redirections.

   run ['scp', '-p', $from_location, $to_location],
     '>', '/dev/null',   # yes, I know /dev/null isn't portable
     '2>', '/dev/null';  # we could probably use IO::Null instead
   

   Disadvantage: a complex module like this has certain limitations (e.g. Windows support is experimental), but I doubt you'll run into any issues here.

I strongly suggest you use IPC::Run.

Answer 3

A few suggestions:

• It's not part of the standard Perl distribution, but Net::SSH2 and Net::SSH2::SFTP are two highly ranked CPAN modules that will do what you want. ssh, scp, and sftp all use the same protocol and the sshd daemon. If you can use scp, you can use sftp. This gives you a very nice Perlish way to copy files from one system to another.
• The quotemeta command will take a string and quote all non-text ASSCII characters for you. It's better than attempting to do the situation yourself with the s/../../ substitution.
• You can use qq(..) and q(...) instead of quotation marks. This will allow you to use quotation marks in your string without having to quote them over and over again. This is especially useful in the system command.

For example:

my $error = system qq(scp $user\@host:"$from_location" "$to_location");

• One more little trick: If the system command is passed a single parameter, and that parameter has shell metacharacters in it, the system command will be passed to the default system shell. However, if you pass the system command a list of items, those items are passed to execvp directly without being passed to the shell.

Passing a_list_ of arguments to system via an array is a great way to avoid problems with file names. Spaces, shell metacharacters, and other shell issues are avoided.

my @command;
push @command, 'scp';
push @command, "$from_user\@$from_host:$from_location",
push @command, "$to_user\@$to_host:$to_location"
my $error = system @command;

Answer 4

use Net::OpenSSH:

my $ssh = Net::OpenSSH->new($host);
$ssh->scp_get($remote_path, $local_path);

The way arguments have to be quoted varies depending on the shell running on the remote side. The stable version of the module has support for Bourne compatible shells. The development version available from GitHub has also support for csh and several Windows flavors (Windows quoteing is, err, interesting).

For instance:

my $ssh = Net::OpenSSH->new($host, remote_shell => 'MSWin', ...);

Note that on Windows, there are string that just can not be properly quoted!