Python-style generators in Go

Question

I'm currently working through the Tour of Go, and I thought that goroutines have been used similarly to Python generators, particularly with Question 66. I thought 66 looked complex, so I rewrote it to this:

package main

import "fmt"

func fibonacci(c chan int) {
    x, y := 1, 1

    for {
        c <- x
        x, y = y, x + y
    }
}

func main() {
    c := make(chan int)
    go fibonacci(c)

    for i := 0; i < 10; i++ {
        fmt.Println(<-c)
    }
}

This seems to work. A couple of questions:

1. If I turn up the buffer size on the channel to say, 10, fibonacci would fill up 10 further spots, as quickly as possible, and main would eat up the spots as quickly as it could go. Is this right? This would be more performant than a buffer size of 1 at the expense of memory, correct?
2. As the channel doesn't get closed by the fibonacci sender, what happens memory-wise when we go out of scope here? My expectation is that once c and go fibonacci is out of scope, the channel and everything on it gets garbage-collected. My gut tells me this is probably not what happens.

Answer 1

I like @tux21b's answer; having the channel created in the fib() function makes the calling code nice and clean. To elaborate a bit, you only need a separate 'quit' channel if there's no way to tell the function when to stop when you call it. If you only ever care about "numbers up to X", you can do this:

package main

import "fmt"

func fib(n int) chan int {
    c := make(chan int)

    go func() {
        x, y := 0, 1

        for x < n {
            c <- x
            x, y = y, x+y
        }

        close(c)
    }()

    return c
}

func main() {
    // Print the Fibonacci numbers less than 500
    for i := range fib(500) {
        fmt.Println(i)
    }
}

If you want the ability to do either, this is a little sloppy, but I personally like it better than testing the condition in the caller and then signalling a quit through a separate channel:

func fib(wanted func (int, int) bool) chan int {
    c := make(chan int)

    go func() {
        x, y := 0, 1

        for i := 0; wanted(i, x); i++{
            c <- x
            x, y = y, x+y
        }

        close(c)
    }()

    return c
}

func main() {
    // Print the first 10 Fibonacci numbers
    for n := range fib(func(i, x int) bool { return i < 10 }) {
        fmt.Println(n)
    }

    // Print the Fibonacci numbers less than 500
    for n := range fib(func(i, x int) bool { return x < 500 }) {
        fmt.Println(n)
    }
}

I think it just depends on the particulars of a given situation whether you:

1. Tell the generator when to stop when you create it by
   1. Passing an explicit number of values to generate
   2. Passing a goal value
   3. Passing a function that determines whether to keep going
2. Give the generator a 'quit' channel, test the values yourself, and tell it to quit when appropriate.

To wrap up and actually answer your questions:

1. Increasing the channel size would help performance due to fewer context switches. In this trivial example, neither performance nor memory consumption are going to be an issue, but in other situations, buffering the channel is often a very good idea. The memory used by make (chan int, 100) hardly seems significant in most cases, but it could easily make a big performance difference.

2. You have an infinite loop in your fibonacci function, so the goroutine running it will run (block on c <- x, in this case) forever. The fact that (once c goes out of scope in the caller) you won't ever again read from the channel you share with it doesn't change that. And as @tux21b pointed out, the channel will never be garbage collected since it's still in use. This has nothing to do with closing the channel (the purpose of which is to let the receiving end of the channel know that no more values will be coming) and everything to do with not returning from your function.

Answer 2

Yes, increasing the buffer size might drastically increase the execution speed of your program, because it will reduce the number of context switches. Goroutines aren't garbage-collected, but channels are. In your example, the fibonacci goroutine will run forever (waiting for another goroutine to read from the channel c), and the channel c will never be destroyed, because the fib-goroutine is still using it.

Here is another, sightly different program, which do not lack memory and is imho more similar to Python's generators:

package main

import "fmt"

func fib(n int) chan int {
    c := make(chan int)
    go func() {
        x, y := 0, 1
        for i := 0; i <= n; i++ {
            c <- x
            x, y = y, x+y
        }
        close(c)
    }()
    return c
}

func main() {
    for i := range fib(10) {
        fmt.Println(i)
    }
}

Alternatively, if you do not know how many Fibonacci numbers you want to generate, you have to use another quit channel so that you can send the generator goroutine a signal when it should stop. This is whats explained in golang's tutorial https://tour.golang.org/concurrency/4.

Answer 3

You could use closures to simulate a generator. Here is the example from golang.org.

package main

import "fmt"

// fib returns a function that returns
// successive Fibonacci numbers.
func fib() func() int {
    a, b := 0, 1
    return func() int {
        a, b = b, a+b
        return a
    }
}

func main() {
    f := fib()
    // Function calls are evaluated left-to-right.
    fmt.Println(f(), f(), f(), f(), f())
}

Answer 4

Using channels to emulate Python generators kind of works, but they introduce concurrency where none is needed, and it adds more complication than's probably needed. Here, just keeping the state explicitly is easier to understand, shorter, and almost certainly more efficient. It makes all your questions about buffer sizes and garbage collection moot.

type fibState struct {
    x, y int
}

func (f *fibState) Pop() int {
    result := f.x
    f.x, f.y = f.y, f.x + f.y
    return result
}

func main() {
    fs := &fibState{1, 1}
    for i := 0; i < 10; i++ {
        fmt.Println(fs.Pop())
    }
}