Stream data into protobuf parser c++

Question

Context

I am writing an event-driven application server in C++. I want to use google protocol buffers for moving my data. Since the server is event driven, the connection handler API is basically a callback function letting me know when another buffer of N bytes has arrived from the client.

Question

My question as a complete beginner of protobuf is this: is it possible to somehow coax protobuf into accepting the many buffers required to make up one complete message to facilitate a "stream parser" rather than waiting for the whole data to arrive into a temporary buffer first?

In other words i want this:

//Event API. May be called multiple times for each protobuf message
bool data_arrived_from_client(void *buf,size_t len){
    my_protobuf.parse_part(buf,len); // THIS IS THE GROSSLY SIMPLIFIED SEMANTIC OF WHAT I WANT
    message_size+=len;
    if(message_size>COMPLETE_BUFFER_SIZE){
        use_complete_protobuf();
        return true;
    }
    return false;
}

..instead of this:

//Event API. May be called multiple times for each protobuf message
bool data_arrived_from_client(void *buf,size_t len){
    my_temp_buffer.append_data(buf,len);
    message_size+=len;
    if(message_size>COMPLETE_BUFFER_SIZE){
        my_protobuf.ParseFromArray(my_temp_buffer.data_ptr(),my_temp_buffer.size());
        use_complete_protobuf();
        return true;
    }
    return false;
}

Answer

Answers with complete code is especially appreciated!

Answer 1

No, this is not possible.

The Protobuf parser is a recursive descent parser, meaning quite a bit of its state is stored on the stack. This makes it fast, but it means there's no way to pause the parser in the middle except to pause the whole thread. If your app is non-blocking, you'll just have to buffer bytes until you have a whole message to parse.

That said, this isn't as bad as it sounds. Remember that the final parsed representation of the message (i.e. the in-memory message object) is much larger than the wire representation. So you are hardly wasting memory on buffering compared to what you're going to do with it later. In fact, holding off on parsing until you actually have all the data may actually save memory, since you aren't holding on to a large half-parsed object that's just sitting there waiting for data to arrive.

Answer 2

Yes, this is possible, I've done it in Javascript, but the design could be ported to C++.

https://github.com/chrisdew/protostream