Tricky stderr redirection with s3cmd

Question

I have some strange behavior redirecting stdout and stderr from s3cmd. For example:

$ touch test

$ s3cmd put test s3://non_existent_bucket
test -> s3://non_existent_bucket/test  [1 of 1]
 4 of 4   100% in    0s    88.17 B/s  done
ERROR: S3 error: 404 (NoSuchBucket): The specified bucket does not exist

This output is (if I'm not getting it wrong) part stdout and part stderr. That is shown as follows:

$ s3cmd put test s3://non_existent_bucket 2> log
test -> s3://non_existent_bucket/test  [1 of 1]
 4 of 4   100% in    0s   120.11 B/s  done

$ cat log
ERROR: S3 error: 404 (NoSuchBucket): The specified bucket does not exist

Effectively, stderr is redirected, but we want both stdout and stderr. Let's try only stdout first:

$ s3cmd put test s3://non_existent_bucket > log
ERROR: S3 error: 404 (NoSuchBucket): The specified bucket does not exist

$ cat log

It is not written after the command so I understand stdout is in fact redirecting (or something), but it doesn't get caught on the log file. I guess it somehow detects if it is connected to a tty or not, and only outputs this kind of data in the first case.

In the same way, if I try to redirect both streams, I get stderr but not the stdout output that I need:

$ s3cmd put test s3://non_existent_bucket >& log

$ cat log
ERROR: S3 error: 404 (NoSuchBucket): The specified bucket does not exist

In fact, what I need to do is call this from PHP and get the full output. I've only found ways to execute a command and retrieve stdout, so I'm doing as follows:

exec('s3cmd put test s3://non_existent_bucket 2>&1', $output);

And here, I don't get any of the output from either streams. I guess I'm loosing stdout because s3cmd detects it's not connected to a tty, but stderr I still don't understand why it's not caught.

Trying with popen() and exec() I get the same result.

I need the whole output, or stdout at least (although I guess the progress info will be somehow grabbled). Any idea?

Thanks in advance!!

Answer 1

The man page for s3cmd details a --progress/--no-progress flag, with a description as follows:

Display or don't display progress meter. When running on TTY (e.g. console or xterm) the default is to display progress meter. If not on TTY (e.g. output is redirected somewhere or running from cron) the default is to not display progress meter.

So it looks like s3cmd will do something like isatty(0) to determine whether to write the progress details to STDOUT or not, but if you explicitly specify --[no-]progress it will force one behaviour or the other.

Try:

exec('s3cmd --progress put test s3://non_existent_bucket 2>&1', $output);