protocol buffers: how to serialize and deserialize multiple messages into a file (c++)?

Question

I am new to Protocol Buffers and c++ but my task requires me to use the two. I want to write a structure of data ( message) into a single file multiple times and be able to read the data. i can read and write a single message but multiple messages is proving harder. I have looked for answers for hours but i can't seem to be able to read the data as a structure. Any example code or pointers will be very helpful.

This is the format of my structure:

    typedef struct Entry
       {
          char  name[ NAME_MAX];
          int   id;
          int   age;
          char  DoB[32]; 
       } entry;

This is what i've been using to write into a file:

    Person_File file;
    fstream output("file", ios::out | ios::trunc | ios::binary);
    file.SerializeToOstream(&output);

I have tried to change the file editing options to append instead of truncate but that does not let me read the data in the format that I want.

this is what I use to read:

    fstream input("file", ios::in | ios::binary);
    file.ParseFromIstream(&input);

These are the contents of my .proto file:

    message Person {


message File {
    required string pname =1;
    required int32 id =2;
    required int32 age =3;
    required string DoB =4;
}

repeated File file =1;

}

From all the searching i've done it seems like CodedInputStream/CodedOutputStream are my best options but I haven't been able to find any detailed examples or explanation to help me understand. I understand that Protocol Buffers are not self delimiting and this is probably the reason why I can't read my messages back in the original format. Any help would be appreciated. Thanks

EDIT: I have tried to use CodedOutputStream based on the messages I've recieved but it doesn't seem to write anything into the file.

    int fd = open("file.txt",O_WRONLY | O_APPEND | O_CREAT);

    FileOutputStream* file_ostream_ = new FileOutputStream(fd);
    CodedOutputStream* ostream_ = new CodedOutputStream(file_ostream_);


ostream_->WriteLittleEndian32(file.ByteSize());
    file.SerializeToCodedStream(ostream_);

After using this code the file comes out blank. Where am I going wrong ?

Answer 1

I have tried to use CodedOutputStream based on the messages I've recieved but it doesn't seem to write anything into the file.

I suspect your problem here is that you aren't deleting the CodedOutputStream nor the FileOutputStream. These objects buffer output and flush the buffer in their destructors, so if you never destroy them, they won't write the last buffer, which in this case is the only buffer.

I recommend allocating these objects on the stack (as local variables). Then you can't possibly forget to destroy them.

With that out of the way, here's code that uses CodedInputStream and CodedOutputStream to read/write delimited messages. Each message is prefixed with a varint indicating the size, which is the same format as the Java Protobuf library's writeDelimitedTo()/parseDelimitedFrom() (a feature that never made it into the C++ library).

bool writeDelimitedTo(
    const google::protobuf::MessageLite& message,
    google::protobuf::io::ZeroCopyOutputStream* rawOutput) {
  google::protobuf::io::CodedOutputStream output(rawOutput);

  // Write the size.
  const int size = message.ByteSize();
  output.WriteVarint32(size);

  uint8_t* buffer = output.GetDirectBufferForNBytesAndAdvance(size);
  if (buffer != NULL) {
    // Optimization:  The message fits in one buffer, so use the faster
    // direct-to-array serialization path.
    message.SerializeWithCachedSizesToArray(buffer);
  } else {
    // Slightly-slower path when the message is multiple buffers.
    message.SerializeWithCachedSizes(&output);
    if (output.HadError()) return false;
  }

  return true;
}

bool readDelimitedFrom(
    google::protobuf::io::ZeroCopyInputStream* rawInput,
    google::protobuf::MessageLite* message) {
  google::protobuf::io::CodedInputStream input(rawInput);

  // Read the size.
  uint32_t size;
  if (!input.ReadVarint32(&size)) return false;

  // Tell the stream not to read beyond that size.
  auto limit = input.PushLimit(size);

  // Parse the message.
  if (!message->MergePartialFromCodedStream(&input)) return false;
  if (!input.ConsumedEntireMessage()) return false;

  // Release the limit.
  input.PopLimit(limit);

  return true;
}