invoking sftp through pipes: the prompt is not returned

Question

I'm trying to copy files using SFTP by calling the SFTP process using pipes (on linux).

I first tried to call "ls" but my problem is that I don't know when the output from the command is finished, so that I know I can send another command, and check the output from the first command.

I would expect sftp to print out its prompt again when the command is done, like so:

sftp>ls
...output...
sftp>

That way I would know when the command is finished. However what happens is that I just get ...output... but I never receive the new sftp> prompt.

I don't understand how come this works in command-line and not when manually ran through pipes. Also this PHP article suggests this approach of waiting for the sftp> prompt.

I have coded two implementations, in java and ruby, and both show the same behaviour. I'm careful of not using a buffered reader, because I'm aware that could cause a problem since sftp does not place a carriage return after the "sftp>" prompt, but to no avail.

If I run in the shell echo ls | sftp <parameters> then it outputs and exits, without giving a new prompt. That would actually be a "good enough" behaviour for me.

I am aware I can write all the commands I want to run in a file and give that file to sftp through a command argument, but I find the pipe approach more workable and less clunky than resorting to a temporary file.

Here is the Java implementation:

String[] cmd = new String[]{"sftp", Config.getSshTargetUsername()
      + "@" + Config.getSshTargetHost()};

Process p = Runtime.getRuntime().exec(cmd);
InputStreamReader inp = new InputStreamReader(p.getInputStream());
OutputStreamWriter out = new OutputStreamWriter(p.getOutputStream());

out.write("ls\n");
out.flush();

StringBuilder read = new StringBuilder();
char[] buf = new char[256];
while (true) {
    int bytes = inp.read(buf);
    read.append(buf, 0, bytes);
    System.out.println(read);
    System.out.flush();
}

And the ruby one:

require 'open3'
cmd = "sftp #{username}@#{host}"
Open3.popen2e(cmd) do |stdin, stdout_err, wait_thr|
  stdin.puts "ls"
  while chr = stdout_err.getc
    putc chr
  end
end

UPDATE: now I realised if I launch bash for instance I get the exact same behaviour. So it's a general thing about pipes, not specifically tied to sftp.

Answer 1

Alright I found the explanation and the correct answer.

In the sftp source code I see this:

if (interactive)
    printf("sftp> ");

And a bit earlier:

interactive = !batchmode && isatty(STDIN_FILENO);

And the isatty() is the key. That way sftp can detect whether it is being executed by a live user, from a tty, or by another application.

So I must make it believe it was launched by a live user. A quick google turned up this link: Trick an application into thinking its stdin is interactive, not a pipe

I used script to launch sftp, and from my tests it appears to work perfectly... So that's the correct way I'd say! About the fact that it appears to work out of the box in PHP, I guess PHP is doing that script trick out of the box when launching external processes.

Answer 2

So for now I gave in (already launching sftp by hand was backing off from using a proper sftp library, I have no more time for this), and I'm now launching directly the solution with pipes. That way sftp exits when it's done processing the command. I launch a new sftp for each operation, but it's workable.

In java:

String[] cmd = new String[] { "/bin/sh", "-c", 
      "echo ls | sftp " + Config.getSshTargetUsername() + "@" +
      Config.getSshTargetHost() };

Process p = Runtime.getRuntime().exec(cmd);
BufferedReader inp = new BufferedReader(new
      InputStreamReader(p.getInputStream()));
String line;
while ((line = inp.readLine()) != null) {
    System.out.println(line);
}

In ruby:

require 'open3'
cmd = 'echo ls | sftp #{username}@#{host}'
Open3.popen2e(cmd) do |stdin, stdout_err, wait_thr|
  while l = stdout_err.readline
    puts l
  end
end

I'm still curious for why my first solution wouldn't work, I hope someone answers it someday.