Is there an effective way to easily get the panic log of a Go program under Unix?

Question

Since I run Go as a server, I need some mechanism to catch the panic log if something goes wrong for later analysis and debugging. Is there an effective way to easily get the panic log of a Go program under Unix? Can you guys introduce your experience? Thanks:)

Answer 1

I get a notification on my phone for some of my fatal panic adventures. Here's how:

Firstly, I usually run everything under daemontools (or the like), so it monitors and reboots on failure.

Then I usually log into syslog using the built-in log package. My syslog is redirected to papertrail , where I can view the status of things, set up alerts, etc. .... Here I send an unwanted notification event to an email address and notifymyandroid so that I can find out about problems, find similar recent problems, look at context of problems, etc ...

... but you cannot register your own unauthorized fatal panic, so I wrote logexec to execute the program and write its stdout and stderr separately, along with an unsuccessful exit notification.

Example:

logexec -tag myprogram /path/to/myprogram -and -its arguments 

Answer 2

You can programmatically catch some problems and process a panic log for them. But this will not work, for example. OOM errors or deadlocks.

A limited case can be illustrated, for example:

 package main import ( "fmt" "os" "runtime" ) func fact(n int) int { if 1/n == 42 { return 314 } return n * fact(n-1) } func main() { const maxTrace = 1000 defer func() { if e := recover(); e != nil { fmt.Fprintf(os.Stderr, "Panic log, recovered: '%v'", e) var b [maxTrace]byte fmt.Fprintf(os.Stderr, "%s", b[:runtime.Stack(b[:], true)]) } }() fact(3) } 

Playground

---

Output:

 Panic log, recovered: 'runtime error: integer divide by zero'goroutine 1 [running]: main.func·001() /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:23 +0x14b main.fact(0x0, 0x4103f1) /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:10 +0x2b main.fact(0x1, 0x0) /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:14 +0x54 main.fact(0x2, 0x0) /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:14 +0x54 main.fact(0x3, 0x401005) /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:14 +0x54 main.main() /tmpfs/gosandbox-9ec179e4_f9a94b22_357f29a2_2f5f25c4_b484febf/prog.go:27 +0x37 

To capture the stack trace for all goroutines during any process failure, including OOM and deadlocks, ...: redirect your stderr to any location (e.g. pipe, file).