How can I properly implement threaded downloads using WebClient with a BlockingCollection?

Question

I'm attempting to make a multi-threaded download manager that has a limit of 4 concurrent downloads. In my research, I came across the following: C# Downloader: should I use Threads, BackgroundWorker or ThreadPool?

[edit] updated code:

Imports System.Net
Imports System.Collections.Concurrent
Imports System.ComponentModel
Imports System.Threading

Public Class Form1

    Const MaxClients As Integer = 4
    ' create a queue that allows the max items
    Dim ClientQueue As New BlockingCollection(Of WebClient)(MaxClients)

    ' queue of urls to be downloaded (unbounded)
    Dim UrlQueue As New Queue(Of String)()

    Dim downloadThread As Thread

    'Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

        ' create four WebClient instances and put them into the queue
        For i As Integer = 0 To MaxClients - 1
            Dim cli = New WebClient()
            AddHandler cli.DownloadFileCompleted, AddressOf DownloadFileCompleted
            AddHandler cli.DownloadProgressChanged, AddressOf DownloadProgressChanged

            ClientQueue.Add(cli)
        Next

        ' Fill the UrlQueue here
        UrlQueue.Enqueue("http://www.gnu.org/licenses/gpl-1.0.txt")
        UrlQueue.Enqueue("http://www.gnu.org/licenses/gpl-2.0.txt")
        UrlQueue.Enqueue("http://www.gnu.org/licenses/gpl-3.0.txt")
        UrlQueue.Enqueue("http://www.gnu.org/licenses/lgpl-2.1.txt")
        UrlQueue.Enqueue("http://www.gnu.org/licenses/lgpl-3.0.txt")
        UrlQueue.Enqueue("http://www.gnu.org/licenses/fdl-1.1.txt")
        UrlQueue.Enqueue("http://www.gnu.org/licenses/fdl-1.2.txt")
        UrlQueue.Enqueue("http://www.gnu.org/licenses/fdl-1.3.txt")

        downloadThread = New Thread(AddressOf downloadQueue)
        downloadThread.IsBackground = True
        downloadThread.Start()
    End Sub

    Private Sub downloadQueue()
        ' Now go until the UrlQueue is empty
        While UrlQueue.Count > 0
            Dim cli As WebClient = ClientQueue.Take() ' blocks if there is no client available

            Dim url As String = UrlQueue.Dequeue()

            Dim fname As String = CreateOutputFilename(url)
            cli.DownloadFileAsync(New Uri(url), fname, New DownloadArgs(url, fname, cli))
            AppendText(url & " started" & vbCrLf)
        End While
    End Sub

    Private Sub DownloadProgressChanged(sender As Object, e As DownloadProgressChangedEventArgs)
        Dim args As DownloadArgs = DirectCast(e.UserState, DownloadArgs)
        ' Do status updates for this download
    End Sub

    Private Sub DownloadFileCompleted(sender As Object, e As AsyncCompletedEventArgs)
        Dim args As DownloadArgs = DirectCast(e.UserState, DownloadArgs)
        ' do whatever UI updates

        Dim url As String = "Filename" '<============I'd like to be able to pass the filename or URL but can't figure this out
        AppendText(url & " completed" & vbCrLf)

        ' now put this client back into the queue
        ClientQueue.Add(args.Client)
    End Sub

    Public Function CreateOutputFilename(ByVal url As String) As String
        Try
            Return url.Substring(url.LastIndexOf("/") + 1)
        Catch ex As Exception
            Return url
        End Try        
    End Function

    Private Delegate Sub SetTextCallback(text As String)

    Private Sub AppendText(text As String)
        If Me.TextBox1.InvokeRequired Then
            TextBox1.Invoke(New Action(Of String)(AddressOf AppendText), text)
            Return
        End If
        Me.TextBox1.AppendText(text)
        Me.TextBox1.SelectionStart = TextBox1.TextLength
        Me.TextBox1.ScrollToCaret()
    End Sub
End Class

Class DownloadArgs
    Public ReadOnly Url As String
    Public ReadOnly Filename As String
    Public ReadOnly Client As WebClient
    Public Sub New(u As String, f As String, c As WebClient)
        Url = u
        Filename = f
        Client = c
    End Sub
End Class

This will successfully download the first 4 files in the UrlQueue, but it then seems to freeze and no further files download. I'd imagine the problem lies in something minor I missed in the process of converting from C# to vb.net, but I can't seem to figure this out.

Answer 1

ClientQueue.Take() blocks the UI thread. Also, WebClient will want to raise the DownloadFileCompleted event on the UI thread - but it is already blocked by ClientQueue.Take(). You have a deadlock.

To resolve this, you got to move your blocking loop to another background thread.

Answer 2

You are blocking the ability for your async queue to process. Not sure this is the "Correct" way to do this but the changes here make it work:

While UrlQueue.Count > 0
    Do While ClientQueue.Count = 0
        Application.DoEvents()
    Loop
    Dim cli As WebClient = ClientQueue.Take() ' blocks if there is no client available
    Dim url As String = UrlQueue.Dequeue()
    Dim fname As String = CreateOutputFilename(url)  ' or however you get the output file name
    cli.DownloadFileAsync(New Uri(url), fname, New DownloadArgs(url, fname, cli))
End While