Approaches for reporting progress for competing consumer scenario

Question

I am getting my head around messaging. Currently we are spiking a few scenarios using Rebus. We are also considering NServiceBus.

The scenario we are trying to build is a proof of concept for a background task processing system. Today we have a handful of backend services hosted in different ways. (web, windows services, console apps) I am looking to hook them up to rebus and start consuming messages using competing consumer, some mesages will have one listener and some will share the load of messages. Elegant :)

I got a pretty good start from this other question How should I set rebus up for one producer and many consumers and it is working nicely in the proof of concept.

Now I want to start reporting progress. My intital approach is to set up pub/sub as well and spin up a service that listens to progress events from all the services. And if a service is interrested in a specific progress in the future it is easy to subscripe of interrest to the messages and start listening.

But how shall I approach setting up both competing consumer and pub/sub? it is dimply two separate things? (In the rebus case one adapter using UseSqlServerInOneWayClientMode / UseSqlServer and another adapter that is set up for the pub/sub using whatever protocol we want?)

Or is there a better solution then having two "buses" here?

Answer 1

I've built something like that myself a couple of times, and I've had pretty good results with using SignalR to report progress from this kind of backend worker processes.

Our setup had a bunch of WPF clients, one single SignalR hub, and a bunch of backend worker processes. All WPF clients and all backend workers would then establish a connection to the hub, allowing workers to send progress reports while doing their work.

SignalR has some nice properties that makes it very suitable for this exact kind of problem:

1. The published messages "escape" the Rebus unit of work, allowing progress report messages to be sent several times from within one single message handler even though it could take a long time to complete
2. It was easy to get the messages all the way to the clients because they subscribe directly
3. We could use the hub groups functionality to group users so we could target progress/status messages from the backend at either all users or a single user (could also be used for departments, etc.)

The most important point, I guess, is that this progress reporting thing (at least in our case) was not as important as our Rebus messages, i.e. it didn't require the same reliability etc, which we could use to our advantage and then pick a technology with some other nice properties that turned out to be cool.