goroutines run out of memory UPD: go error handling

Question

UPD: Turns out, it is a question about error handling in Go

I have written a simple web crawler, that generates the addresses of web-pages on the "main thread", fetches the actual pages in one go-routine and gives the content back via a chan, and writes it to a file in another go-routine. However, after about 300,000 iterations of this process, i get the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x38 pc=0x22e9]</pre>

The error "stacktrace" is quite long, and unfotunately, I don't have it now (I will post it later, after repeating the experiment).

Is there a need to somehow manage the memory, close the channels or something like that?

Below is the code with some omissions.

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "strconv"
)

func main() {
    channel := make(chan []byte)

    for i:=0; i < 20; i++ {
        go fetcher(generateLink(), channel)
    }

    for a:=0; ; a++ {
        go writeToFile(strconv.Itoa(a), <-channel)
        go fetcher(generateLink(), channel)
        fmt.Println(strconv.Itoa(a))
    }
}

func fetcher(url string, channel chan []byte) {

    resp, err := http.Get(url)
    if err != nil {
        channel <- []byte("")
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    channel <- body
}

func writeToFile(filename string, bytes []byte) {
    ioutil.WriteFile(filename+".html", bytes, 0644)
}

func generateLink() string {
    ...
}

Answer 1

panic: runtime error: invalid memory address or nil pointer deference

This means you tried to use a nil pointer. This is a programmer error, not an out of memory error. Since I don't have access to the actual trace or code, there is no way I can help you further.

Edit: After taking another look at your code I think I found a possible source of the error.

resp, err := http.Get(url)
if err != nil {
    channel <- []byte("")
}
defer resp.Body.Close()

You end up referencing resp in order to get the field Body even when err != nil. If err !=nil, it is quite likely resp is nil.

Answer 2

I'm only guessing here since I don't have the stacktrace. but the following two lines look suspicious to me:

body, err := ioutil.ReadAll(resp.Body)
channel <- body

How do you know the body is not nil when you send it on the channel? Since writeToFile doesn't check it either it's totally possible that you are tring to write a nil byte slice to the file and it's blowing up there. If you capture an err you should check it before going on.

Answer 3

At some point, your call to http.Get() fails and returns a non-nil err. When this happens, you put an empty byte slice into the channel, but you keep going, and try to read from resp.Body anyway, when resp is nil.