Is there any efficient way to get panic log of Go program under Unix easily?

Question

Since I'm running a Go program as server, I need some mechanism to catch panic log if anything goes wrong for later analyze & debug. Is there any efficient way to get panic log of Go program under Unix easily? Can you guys introduce your experience about this? Thanks :)

Answer 1

I get a notification on my phone for some of my fatal panics on go programs. Here's how:

First, I generally run everything under daemontools (or similar) so it's monitored and will restart itself on failure.

Then, I generally log to syslog using the built-in log package. My syslog forwards to papertrail where I can review the state of things, set up alerts, etc... It is here I forward undesirable event notifications to an email address and notifymyandroid so I can be made aware of problems, search for similar recent issues, look at context around the problems, etc...

...but you can't log your own uncaught fatal panic, so I wrote logexec to execute the program and log its stdout and stderr separately along with an unsuccessful exit notification.

Example:

logexec -tag myprogram /path/to/myprogram -and -its arguments

Answer 2

It's possible to programmatically catch some troubles and process a panic log for them. But it'll not work for eg. OOM errors or deadlocks.

The limited case can be illustrated like, for example:

package main

import (
        "fmt"
        "os"
        "runtime"
)

func fact(n int) int {
        if 1/n == 42 {
                return 314
        }

        return n * fact(n-1)
}

func main() {
        const maxTrace = 1000
        defer func() {
                if e := recover(); e != nil {
                        fmt.Fprintf(os.Stderr, "Panic log, recovered: '%v'", e)
                        var b [maxTrace]byte
                        fmt.Fprintf(os.Stderr, "%s", b[:runtime.Stack(b[:], true)])
                }
        }()

        fact(3)
}

Playground

---

Output:

Panic log, recovered: 'runtime error: integer divide by zero'goroutine 1 [running]:
main.func·001()
        /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:23 +0x14b
main.fact(0x0, 0x4103f1)
        /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:10 +0x2b
main.fact(0x1, 0x0)
        /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:14 +0x54
main.fact(0x2, 0x0)
        /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:14 +0x54
main.fact(0x3, 0x401005)
        /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:14 +0x54
main.main()
        /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:27 +0x37

To capture the stack trace for all goroutines on any kind of process crash, including OOM and deadlocks, ...: redirect its stderr to any desired place (eg. pipe, file).