Arbitrary protobuf message as byte-array over Websocket - how to determine actual message type upfront

Question

Protobuf is nice and dandy, but it was not made with self-description in mind. Now this is totally okay if you're using a well-defined protocol and want to replace e.g. SOAP-messages or just want to replace a restful JSON API or the like.

However I would like to use it over a Websocket though and instead of closing and reopening it up again and again under different urls - which defeats the point of the websocket - I would like to keep it open and send different messages over the wire as Byte-arrays!

Now my problem is simple yet a bit complex to solve. I want to send an arbitrary message from client to the server (or vice versa) and the receiver should simple determine which type of message it got and how to interpret it.

In pseudocode this would look like so:

client:

Message m = new Auth().withUserName("Sorona").withPassword("TotallyNotMyActualPassword")

ws.send(m)

server:

Map[Type, Handler]

receiveMessage(Message m) {
   handlers.get(m.determineType()).handle(m)
}

handler:

trait Handler[T] {
  def handle()
}

So what I truly want is two things:

1) Be able to determine the actual type of Any message without (dramatically) increasing the message size.

2) Be robust and extendable

For a new message type I simply want to add a handler and be done.

No hardcoded enums I have to maintain, no switch-case structures I have to adopt etc.

In simple OOP this would be fairly easy but with Protobufs I am kind of stuck.

Any hints?

Answer 1

What I came up with so far:

message Root
{
    string type = 1;
    google.protobuf.Any content = 2; 
}

message Dog
{
   string name = 1;
   string fav_food = 2;
}

message Cat
{
   string name = 1;
   uint32 colors = 2; 
}

Then somewhere (shared file, database etc.) I create a map like this:

val mapper: Map[String, Class] = Map(
  "Dog" -> Dog,
  "Cat" -> Cat
)

On the sender I would do:

val nero = Dog().withName("Nero").withFavoriteFood("Raw meat")
val msg = Root().withType("Dog").withMessage(Any.pack(nero))

and on the receiver side:

val r = Root.parseFrom(msg)
val a = r.message.get.unpack(mapper(r.type))

That's basically the plan. I would then introduce my own builders that have the types (e.g. "Dog" or "Cat") hardcoded and would also add another mapper between a class/messagetype and a handler.

But I guess if I come that far and it works (currently it doesn't because I am using Scala.js and it looks like the Any.pack is not supported there right now) it will be a totally different story ;)

Answer 2

I can think of different approaches here:

• self describing messages (like in your proposal)
• message description outside of messages (meta-data)
• rpc framework (recommended for complex projects)

description in metadata

If you don't want to put a self describing property into each message, you could define them outside, and for example send them in between messages. For example

stream: ... "now comes a dog", { dog }, "now comes a cat", { cat }, ...

how you encode "now comes a dog" and how you map that to your message is up to you. Depending on how you structure your identifiers, this might be more complicated then having self describing message (if you use strings of arbritary length for example, you would also need to send along that number).

self describing messages

your custom protocol looks like this

stream: ..., { self-describing dog }, { self-describing cat }, ...  

using an rpc framework

you could also try to use an rpc framework like gRPC which will give you a more sophisticated design. Here you can define service interfaces with different methods and parameters. The communication will be generated and you only need to provide the method implementations. However I am not sure if websockets are fully supported