Why does this concurrent HTTP client randomly crash?

Question

I'm learning Go by writing an HTTP testing client like Apache's ab. The code below seems pretty straightforward: I create a configurable number of goroutines, each of which sends a portion of the overall HTTP requests and records the result. I iterate over the resultChan channel and inspect/record each result. This works find when the number of messages is, say, 100. When I increase the number of messages, however, it hangs and htop shows VIRT of 138G for the process.

Here's the code in question:

package main

import "net/http"
import "fmt"
import "time"

const (
    SUCCESS   = iota
    TOTAL = iota
    TIMEOUT = iota
    ERROR = iota
)

type Result struct {
    successful    int
    total         int
    timeouts      int
    errors        int
    duration      time.Duration
}

func makeRequests(url string, messages int, resultChan chan<- *http.Response) {
    for i := 0; i < messages; i++ {
        resp, _ := http.Get(url)
        if resp != nil {
            resultChan <- resp
        }
    }
}

func deployRequests(url string, threads int, messages int) *Result {
    results := new (Result)
    resultChan := make(chan *http.Response)
    start := time.Now()
    defer func() {
        fmt.Printf("%s\n", time.Since(start))
    }()
    for i := 0; i < threads; i++ {
        go makeRequests(url, (messages/threads) + 1, resultChan)
    }

    for response := range resultChan {
        if response.StatusCode != 200 {
            results.errors += 1
        } else {
            results.successful += 1
        }
        results.total += 1
        if results.total == messages {
            return results
        }
    }
    return results
}

func main () {
    results := deployRequests("http://www.google.com", 10, 1000)
    fmt.Printf("Total: %d\n", results.total)
    fmt.Printf("Successful: %d\n", results.successful)
    fmt.Printf("Error: %d\n", results.errors)
    fmt.Printf("Timeouts: %d\n", results.timeouts)
    fmt.Printf("%s", results.duration)
}

There are obviously some things missing or stupidly done (no timeout checking, channel is synchronous, etc) but I wanted to get the basic case working before fixing those. What is it about the program as written that causes so much memory allocation?

As far as I can tell, there are just 10 goroutines. If one is created per HTTP request, which would make sense, how does one perform operations that would create many goroutines in a loop? Or is the issue totally unrelated.

Answer 1

I think the sequence leading to the hang is:

1. http.Get in makeRequests fails (connection denied, request timeout, etc.), returning a nil response and an error value
2. The error is ignored and makeRequests moves on to the next request
3. If any errors occur, makeRequests posts less than the expected number of results to resultChan
4. The for .. range .. chan loop in deployRequests never breaks because results.total is always less than messages

One workaround would be:

If http.Get returns an error value, post a nil response to resultChan:

    resp, err := http.Get(url)
    if err != nil {
        resultChan <- nil
    } else if resp != nil {
        resultChan <- resp
    }

In deployRequests, if the for loop reads a nil value from resultChan, count that as an error:

for response := range resultChan {
    if response == nil {
        results.errors += 1
    } else if response.StatusCode != 200 {

    // ...