boost.asio : multi ports application-layer protocol

Question

As has been pointed out, any system that's required to be robust should not assume that routing/firewalling is such that connections can be made back to the client from the server.

How would we go around this?

client connects to server:8000
server responds with a session UID
client is responsible for connecting two more "channels", using the session UID as a correlation ID
server decides that a session is completely ready if all 3 "channels" (i.e. sockets) participating in with a given session UID have been connected. If there's a timeout for this to happen, the server jots the whole session and closes any sockets that had been opened.

This is fairly easy, but requires a wire protocol to coordinate the sessions and channel "roles" for connections. I did initially elect not to implement this as a demo. Instead I thought it would be a nice exercise to learn Boost Asio, and implemented the back-channels (initiated from the server side) as in your original drawing.

The full code is on github: https://gist.github.com/sehe/9946161

Notes:

there is a 'general' listener implementation that is used for both the server (port 8000) and "back-channels" (ports 8001,8002). See listener.hpp
I opted for the stackless coroutine approach. This requires Boost Asio 1.54
This approach has led to relative abuse of shared_ptr<>, at least in my view. The reason is that it's very beneficial if the coroutine (which is also the completion functor) is copyable without any issues. I could probably clean this up by making the class itself enable_shared_from_this and bind to shared_from_this instead. The benefit now is that there are (almost) no bind-expressions.

Creating the 'back channels' is done in the server class which overrides on_accept:

virtual bool on_accept(tcp::socket& socket) override
{
    auto host = socket.remote_endpoint().address().to_string();
    // for now setting up the back-connections is all synchronous -
    // that might not work well in practice (scaling, latency) but...
    try
    {
        tcp::resolver resolver(socket.get_io_service());
        auto ep1 = resolver.resolve(tcp::resolver::query(host, "8001"));
        auto ep2 = resolver.resolve(tcp::resolver::query(host, "8002"));

        backsock1 = make_shared<tcp::socket>(socket.get_io_service());
        backsock2 = make_shared<tcp::socket>(socket.get_io_service());

        backsock1->connect(*ep1);
        backsock2->connect(*ep2);

        std::cerr << "on_accept: back channels connected for " << host << "\n";
    } catch(std::exception const& e)
    {
        std::cerr << "on_accept: '" << e.what() << "' for " << host << "\n";
        return false;
    }

    return base_type::on_accept(socket);
}

If on_accept fails (e.g. for our server cannot connect the back-channels) an error is returned on the "main" socket (initial connection) and the session is aborted

There are three programs:

run_server (which listens on port 8000)
run_client (which connects to port 8000 and listens on 8001,8002), and sends 1 message. You can observe how the the server responds by connecting on the back-channels and sending different messages on all three sockets.

test (which combines the two):

#include <boost/asio.hpp>
#include <boost/thread.hpp>
#include "server.hpp"
#include "client.hpp"

int main()
{
    boost::asio::io_service svc;

    // start service on a separate thread
    boost::thread th([&svc] { 
            svc.post(demo::server(svc));
            svc.run(); 
            });

    boost::this_thread::sleep_for(boost::chrono::milliseconds(500)); // allow server to start accepting

    // post client traffic to the service
    std::cerr << "Starting a test client that sends a message...\n";
    demo::client client(svc, "localhost", "8000");

    // await interrupt (or new connections)
    th.join();
}

The output of the last program looks like:

Starting a test client that sends a message...
on_accept: back channels connected for 127.0.0.1
listener 127.0.0.1:8000: accepting connection from 127.0.0.1:40999
listener 127.0.0.1:8000: 'hello world from demo client' received from 127.0.0.1:40999
listener 127.0.0.1:8001: accepting connection from 127.0.0.1:40132
listener 127.0.0.1:8002: accepting connection from 127.0.0.1:37970
listener 127.0.0.1:8001: 'We've received a request of length 29' received from 127.0.0.1:40132
listener 127.0.0.1:8002: 'We're handling it in void demo::server::do_back_chatter(const string&)' received from 127.0.0.1:37970
listener 127.0.0.1:40999: 'ECHO hello world from demo client' received from 127.0.0.1:8000