pg_ctl hangs over ssh

Question

I am designing a process to test my postgresql 10.8 backups by restoring them to a random point in time in a throwaway virtual machine. I have been unable to fully automate the process though. I am blocked at step 8 (section 25.3.4) of the official documentation

8. Start the server.

When executing the pg_ctl start over ssh, the command hangs until it is killed. If I ssh directly onto the VM and execute pg_ctl start, then the command returns quickly as expected.

This thread from 2012 seems to describe a similar scenario. In my case however the postgres process does start successfully even if the calling session is killed while it is hanging (possibly an improvement between 9.0.5 and 10.8 though?).

This github issue seems relevant but sadly is "solved" by a long rewrite in a language I don't know and concludes finally that it's a bug in the pg_ctl binary.

The Big Question

How can I automate step 8 so that I can continue on to subsequent validation tests of my backup media?

Is this an outstanding bug in the binary I need to hack around? Or have I missed a sensible implementation?

Answer 1

It looks like pg_ctl wants a real terminal for output, which is not allocated by ssh when you simply ask it to run a command. According to the Postgres manual

On Unix-like systems, by default, the server's standard output and standard error are sent to pg_ctl's standard output (not standard error). The standard output of pg_ctl should then be redirected to a file or piped to another process such as a log rotating program like rotatelogs; otherwise postgres will write its output to the controlling terminal (from the background) and will not leave the shell's process group. [...] These default behaviors can be changed by using -l to append the server's output to a log file. Use of either -l or output redirection is recommended.

Therefore, you either need to tell ssh to allocate a terminal for pg_ctl using the -t options (ssh -t somehost "pg_ctl start") or tell pg_ctl to not write to the terminal (ssh somehost "pg_ctl start >/dev/null" or ssh somehost "pg_ctl start -l /tmp/start.log ")