How to measure system overload when using GO

Question

I am rewriting an old system in GO, and in the old system I was measuring the system load average to know if I should increase the number of thread in my thread-pool.

In go people are not using threadpool or pool of goroutine because starting a goroutine is very cheap. But still running too many goroutine is less efficient then just enough to keep the cpu usage near 100%

Thus is there a way to know how many goroutine are ready to run (not blocked) but not currently running. Or is there a way to get the number of scheduled runnable goroutine "Run queue".

Answer 1

Check out the runtime/pprof package.

To print "stack traces of all current goroutines" use:

pprof.Lookup("goroutine").WriteTo(os.Stdout, 1)

To print "stack traces that led to blocking on synchronization primitives" use:

pprof.Lookup("block").WriteTo(os.Stdout, 1)

You can combine these with the functions in the runtime package such as runtime.NumGoroutine to get some basic reporting.

This example deliberately creates many blocked goroutines and waits for them to complete. Every 5 seconds it prints the output of the block pprof profile, as well as the number of goroutines still in existence:

package main

import (
    "fmt"
    "math/rand"
    "os"
    "runtime"
    "runtime/pprof"
    "strconv"
    "sync"
    "time"
)

var (
    wg sync.WaitGroup
    m  sync.Mutex
)

func randWait() {
    defer wg.Done()
    m.Lock()
    defer m.Unlock()
    interval, err := time.ParseDuration(strconv.Itoa(rand.Intn(499)+1) + "ms")
    if err != nil {
        fmt.Errorf("%s\n", err)
    }
    time.Sleep(interval)
    return
}

func blockStats() {
    for {
        pprof.Lookup("block").WriteTo(os.Stdout, 1)
        fmt.Println("# Goroutines:", runtime.NumGoroutine())
        time.Sleep(5 * time.Second)
    }
}

func main() {
    rand.Seed(time.Now().Unix())
    runtime.SetBlockProfileRate(1)
    fmt.Println("Running...")
    for i := 0; i < 100; i++ {
        wg.Add(1)
        go randWait()
    }
    go blockStats()
    wg.Wait()
    fmt.Println("Finished.")
}

I'm not sure if that's what you're after, but you may be able to modify it to suit your needs.

Playground

Answer 2

is there a way to know how many goroutine are ready to run (not blocked) but not currently running.?

You will be able (Q4 2014/Q1 2015) to try and visualize those goroutines, with a new tracer being developed (Q4 2014): Go Execution Tracer

The trace contains:

• events related to goroutine scheduling:
  • a goroutine starts executing on a processor,
  • a goroutine blocks on a synchronization primitive,
  • a goroutine creates or unblocks another goroutine;
• network-related events:
  • a goroutine blocks on network IO,
  • a goroutine is unblocked on network IO;
• syscalls-related events:
  • a goroutine enters into syscall,
  • a goroutine returns from syscall;
• garbage-collector-related events:
  • GC start/stop,
  • concurrent sweep start/stop; and
• user events.

By "processor" I mean a logical processor, unit of GOMAXPROCS.
Each event contains event id, a precise timestamp, OS thread id, processor id, goroutine id, stack trace and other relevant information (e.g. unblocked goroutine id).