maximum throughput in distributed processing (with netty 4.0)

Question

We build system for distributed processing and want to use netty (4.0) for the network I/O stack.

The following situation: We got a producer task A and a consumer task B. Task A produces data in 64K chunks and transmits it to task B. Task B can be in certain circumstances compute intensive and consume the 64K blocks slower than produced by task A. Task A and B are connected by a tcp channel.

We think about this approach: Task A produces the chunks and put it in a local queue. A chunk is automatically taken from the queue, when the tcp channel is free and the next 64K can be written into the channel (does netty give us such a signal/event?). If the queue on Task A exceeds a fixed limit of stored chunks, we block task A until task B have consumed chunks. In essence, we want a "receiver triggered write" for task A to fully utilize the tcp channel without congesting it. The goal of this design should be maximum data throughput.

There are several questions now :)

Is that a good design for enabling maximum throughput? What would be a better design to fully utilize the tcp channel?

Is netty the right framework for these scenarios? (I am pretty new to netty, but I really like the clean abstractions/designs of the framework!)

Can such a design be realized with netty? => (does it give us such a signal/event from the receiver site?)

What would be the best design with netty to enable maximum throughput?

Are there other framework which are more suitable?

Any idea and note is welcome!!!

Many thanks in advance!!! Tobi

Answer 1

here is a few remarks:

• general design : you're mentioning a "distributed" processing, but do not specify if you would have multiple instances of Task A and Task B processor. If you just have one Task A processor and one Task B processor, the data throughput will be determined by the slower one. I understand that, one average, B is slower, but it can be faster, so introducing a buffer between A and B looks like a good idea. So I think the design is fine if you want/need to stick to a single A and a single B instance, but that you might consider having multiple instances if B is indeed slower (you would then have more B instances than A)

• netty or other frameworks : yes you could do that in Netty. However, I think you would have to write the "TCP channel is free" signal. I don't have much experience, but I would think that frameworks like http://akka.io/, implementing message passing and actor model would be interesting to look at.