Splitting work-queue among multiple processes

Question

I have a situation where I believe a work-queue and a SqlDependency would be a good fit, but I'm having trouble piecing it all together.

Let's say I have 1000 entities and each entity must be "processed" at different intervals (i.e. 15, 30 or 45 minutes). I want to split up the work among multiple processes without having to poll the SQL Server every minute looking for more work. I've read a lot of articles related to SqlDependency / Query notification but can't figure out how to queue the work and how to limit each process to a set of work-items.

I wrote a SqlDependency query that looks like the following:

select f1, f2, etc from dbo.entity where LastRunDt < datediff(minute, 15, @dt)

The query runs with 0 results, but when 15 minutes have passed which would cause this recordset to change (more items in it), the notification is not fired. I'm guessing it won't work this way, so I'm stuck on how to get the work items queued.

Also.. when I attached 2 processes to the same work-queue, each process is notified (when I run select * from dbo.entity SqlDependency query and change a record) but they both receive the same work items. I need to partition the work items among the available processes without knowing how many listeners there are, could be 2 or 12 depending on which servers are online and which ones are offline for maintenance.

Clients will be written in C#.

Any known patterns, thoughts or direction would be very much appreciated.

Answer 1

You do not need to involve Query Notification here. Use Service Broker directly.

Have your C# application issue WAITFOR(RECEIVE ...) statements to get work tasks. When work is available, enqueue a message using SEND. To schedule work in future use BEGIN CONVERSATION TIMER.

Additionally Service Broker can help you actually activate your C# code, either as an internal managed stored procedure (see Service Broker Activation) or as external standalone application (see Service Broker External Activator).

Answer 2

Sounds like you need some kind of message queueing. There are some common middleware packages like NServiceBus.

You could also use ServiceBroker which integrates nicely with SQL Server in a transactional and fast way.