Could MSMQ resolve performance bottleneck of out multithreaded services?

Question

We wrote service that using ~200 threads .

200 Threads must do:

1- Download from internet

2- Parse the raw data (html,xml,json...)

3- Store the newly created data to db

For ~10 threads elapsed time for second operation(Parsing) is 50ms (per thread)

For ~50 threads elapsed time for second operation(Parsing) is 80-18000 ms (per thread)

So we have an idea !

We can download documents as multithreaded but using MSMQ we can send rawdata to another process (consumer). And another process implement second part (Parsing) as single threaded.

You can say why dont you use c# Queue class in same process.. We could not prevent our "precious parsing thread" from Thread Context switch. If there are 200 threads in same process the precious will be context switch victim.

Using MSMQ for this requirement is normal?

Answer 1

Yes, this is an excellent example of where MSMQ makes a lot of sense. You can offload your difficult work to a different process to handle without affecting the performance of your current process which clearly doesn't care about the results. Not only that, but if your new worker process goes down, the queue will preserve state and messages (other than maybe the one being worked on when it went down) will not be lost.

Depending on your needs and goals I'd consider offloading the download to the other process as well - passing URLs to work on to the queue for example. Then, scaling up your system is as easy as dialing up the queue receivers, since queue messages are received in a thread safe manner when implemented correctly.

Answer 2

Yes, it is normal. And there are frameworks/libraries that help you building these kind of solutions providing you more than only transports.

NServiceBus or MassTransit are examples (both can sit on top of MSMQ)