Netty nested pipelines / multiplexing

Question

I'm pretty new to Netty, but how would one implement a case in Netty 4.x when several protocols (e.g. P1 and P2) are encapsulated inside another protocol?

              +-------------+
              |   decoder   |
              +-------------+
              |   encoder   |
              +-------------+
              |    muxer    |
              +-------------+
              |   demuxer   |
              +---+------+--+
                  |      |
                  |      |
           +------+      +------+
           |                    |
           |                    |
           v                    v
    +-------------+      +-------------+
    | P1 decoder  |      | P2 decoder  |
    +-------------+      +-------------+
    | P1 encoder  |      | P2 encoder  |
    +-------------+      +-------------+
    | P1 handler  |      | P2 handler  |
    +-------------+      +-------------+

Is there a way to create nested pipelines, so that decoder<->encoder<->muxer<->demuxer being the main pipeline would send the data along P1 or P2 pipeline based on the decision of demuxer?

Or maybe there is a way to somehow create (for the sake of clarity) "subchannels" with their own pipelines?

Answer 1

There is no support for "nested Pipelines" yet. It may be part of 4.1.0. For now you need to remove/add handlers on the fly.

See [1] for an example.

[1] https://github.com/netty/netty/blob/master/example/src/main/java/io/netty/example/portunification/PortUnificationServerHandler.java

Answer 2

I added the following comment to https://github.com/netty/netty/issues/8544

I have a suggestion how this can probably be achieved in current and older versions of netty.

The basic idea is to define a single 'front-end' pipeline to handle the real socket and I/O including encode/decode. Then distinct 'back-end' pipelines will be invoked for various related protocols (or even separate 'sessions' for protocols where the application layer can run independent sessions over a single connection). Thus the back-end pipelines are protocol or session/dialog specific, the front-end is more general and maps to the single real network connection downstream and knows how to instantiate, retrieve and select from multiple back-end channels upstream.

I see two potential levels of abstraction in Netty where this distinction between front-end and back-end pipelines might be possible:

1. The most feasible (and heavyweight) is to declare a whole other Bootstrap for each backend pipeline. These backend bootstraps will have to be bound to a custom 'virtual socket' which is actually backed by the 'front-end' netty pipeline. In other words, the virtual socket is not mapped directly to a TCP connection or host/port, but rather is a further endpoint distinction (host, port AND protocol or session), but it maintains a reference back to the single specific 'front-end' channel/pipeline that invoked it.

2. A less heavy-weight option might be to somehow instantiate (and store and retrieve) new channels within the existing bootstrap. I have hardly begun to investigate how feasible this is. The following questions (and possibly others I haven't thought of) must be answered first:

• Can new channels be instantiated within a pipeline, but added to an event executor?
• Should the back-end channels run in a separate event executor pool?
• How to instantiate, store and retrieve instances of back-end channels from the front-end
• Backend channels must still make use of the same virtual socket concept as option 1
• Can channels/channel pipelines hand-off to other channels?
• Can those back-end channels then write back to the same front-end channel that invoked them? The front-end channel would have to pass a call-back (or a reference to itself) to the back-end channels. Presumably this reference should be managed/held by the virtual socket.