You logically have a linear pipeline, so I think that's how you should model it in code too. This means having a separate download block for each type of pages. This way, completion will work fine, but you'll have to deal with connection limiting separately.
There are two ways I can see how to solve that:
If you're always connecting to the same server, you can limit the number of connections to it by using
ServicePoint
s. You can either set that limit globally at the start of the program:ServicePointManager.DefaultConnectionLimit = limit;
or just for the one server:
ServicePointManager.FindServicePoint(new Uri("http://myserver.com")) .ConnectionLimit = limit;
If using
ServicePoint
s won't work for you (because you don't have just one server, because it affects the whole application, …), you can limit the requests manually using something likeSemaphoreSlim
. The semaphore would be set to your desired limit and it would be shared between the two download blocks.MaxDegreeOfParallelism
for each block would be set to the same limit (higher value won't add anything, lower value could be inefficient) and their code could look like this:try { await semaphore.WaitAsync(); // perform the download } finally { semaphore.Release(); }
If you do need this kind of limiting often, you could create a helper class that encapsulates this logic. Its usage could look like this:
var factory = new SharedLimitBlockFactory<Input, Output>( limit, input => Download(input)); var downloadBlock1 = factory.CreateBlock(); var downloadBlock2 = factory.CreateBlock();