TPL dataflow and BatchJoinBlock with different batch sizes

Question

Let's say I have some data coming from 2 different sources at 2 different frequencies and I have to join them and process them together. Frequency on channel A is 100Hz (100 integers per second), while frequency on channel B is 1Hz (1 integer per second). The data is already processed on each individual channel using ActionBlock from TPL dataflow. Now though, I have to process it together (1000 int from A and 5 int from B at a time). For me the solution would be to join these blocks using BatchJoinBlock that would wait for 1000 messages from channel A (1000 integers) and 5 messages from channel B (5 integers). After batches are populated, fire it to an action block and process.

The problem is - according to the documentation, the BatchJoinBlock (MSDN) would always have all batches of the same size.

How could I go around different sizes?

As an alternative I probably could create 2 different BatchBlock objects with different sizes and then fire them to a JoinBlock, which in turn would send data to an ActionBlock; but that seems like an overkill, or is it a common approach?

I am not posting any code because for now these are purely theoretical approaches I am considering, before I go about an implementation of that join of two different channels. I can quickly knock-up some example if needed though.

Answer 1

You've pretty much figured it all out by yourself. BatchJoinBlock is for cases where you have two types and you want batches of specific size, counting items of both types towards the same size.

An example would be if one type were successes and the second type would be failures. If you wanted to create batches of n outcomes, no matter how many of them are successes and how many are failures, you would use BatchJoinBlock.

I believe there reason why there isn't a single block that would do exactly what you're asking is because you can build the same functionality from two BatchBlocks and a JoinBlock. The strength of Dataflow is in combining various blocks, so I think that's exactly what you should do. (You can't build BatchJoinBlock from other existing blocks.)