You seem to be using two forms of concurrency control in this code, and both have problems.
You've got limitChan
, which looks like it is being used as a semaphore (request
sends a value at its start, and receives a value in a defer
in that function). But checkUrls
is also trying to make sure it only has threadLimit
goroutines running at once (by spawning that number first up, and only spawning more when one reports its results on the ok
channel). Only one of these should be necessary to limit the concurrency.
Both methods fail due to the way the defer
is set up in request
. There are a number of return
statements that occur before defer
, so it is possible for the function to complete without sending the result to the ok
channel, and without freeing up its slot in limitChan
. After a sufficient number of errors, checkUrls
will stop spawning new goroutines and you'll see your hang.
The fix is to place the defer
statement before any of the return
statements so you know it will always be run. Something like this:
func request(url string, threadLimit chan int, ok chan requestReturn, kws string) {
threadLimit <- 1
rr := requestReturn{url: url}
var resp *http.Response
defer func() {
if resp != nil {
resp.Body.Close()
}
ok <- rr
<-threadLimit
}()
...
}