Wait for the termination of n goroutines

Question

I need to start a huge amount of goroutines and wait for their termination. The intuitive way seems to use a channel to wait till all of them are finished :

package main

type Object struct {
    //data
}

func (obj *Object) Update(channel chan int) {
    //update data
    channel <- 1
    return
}

func main() {

    channel := make(chan int, n)
    list := make([]Object, n, m)
    for {
        for _, object := range list {
            go object.Update(channel)
        }
        for i := 0; i < n; i++ {
            <-channel
        }
        //now everything has been updated. start again
    }
}

But the problem is that the amount of objects and therefore the amount of goroutines could change. Is it possible to change the buffer size of a channel ?

Is there maybe a more elegant way to do this ?

Answer 1

I've used WaitGroup as a solution to this problem. Translating your current code, with some logs to make it clear what's happening:

package main

import "sync"
import "fmt"
import "time"

type Object struct {
    //data
}

func (obj *Object) Update(wg *sync.WaitGroup) {
    //update data
    time.Sleep(time.Second)
    fmt.Println("Update done")
    wg.Done()
    return
}

func main() {
    var wg sync.WaitGroup
    list := make([]Object, 5)
    for {
        for _, object := range list {
            wg.Add(1)
            go object.Update(&wg)
        }
        //now everything has been updated. start again
        wg.Wait()
        fmt.Println("Group done")
    }
}

Answer 2

This task in not exactly trivial, it's quite easy to write a buggy one. I recommend to use a ready made solution in the stdlib - sync.WaitGroup. Quoting from the link:

A WaitGroup waits for a collection of goroutines to finish. The main goroutine calls Add to set the number of goroutines to wait for. Then each of the goroutines runs and calls Done when finished. At the same time, Wait can be used to block until all goroutines have finished.

Answer 3

@tjameson did a great job explaining how to use WaitGroup, how to pass a reference to your WaitGroup object to your function. The one change I would make to his example is leverage defer when you are Done. I think this defer ws.Done() should be the first statement in your function.

I like WaitGroup's simplicity. However, I do not like that we need to pass the reference to the goroutine because that would mean that the concurrency logic would be mixed with your business logic.

So I came up with this generic function to solve this problem for me:

// Parallelize parallelizes the function calls
func Parallelize(functions ...func()) {
    var waitGroup sync.WaitGroup
    waitGroup.Add(len(functions))

    defer waitGroup.Wait()

    for _, function := range functions {
        go func(copy func()) {
            defer waitGroup.Done()
            copy()
        }(function)
    }
}

So your example could be solved this way:

type Object struct {
    //data
}

func (obj *Object) Update() {
    //update data
    time.Sleep(time.Second)
    fmt.Println("Update done")
    return
}

func main() {
    functions := []func(){}
    list := make([]Object, 5)
    for _, object := range list {
        function := func(obj Object){ object.Update() }(object)
        functions = append(functions, function)
    }

    Parallelize(functions...)        

    fmt.Println("Group done")
}

If you would like to use it, you can find it here https://github.com/shomali11/util