Protobuf builder: set string field without memory allocation

Question

Let's say I have to create and write huge amount of protobuf messages, which contain string field (different for each message). I create a protobuf.Builder and reuse it for creating all messages, setting this string field and invoking method build(). I have bytes as input, e.g. for each message I have byte[], which should be set to string-field. So the question is - how can I set byte[] to string field of the builder without new memory allocation?

As far as I understand, when use protobuf builder in java there are two ways to set string-field:

1. set(String field)
2. set(ByteString field)

The first one requires to make a new object String from byte[], and that's why new memory allocation. The second one requires to use some method of ByteString, for example ByteString.copyFrom(byte[]), but this method (as all other similar methods of ByteString) does not use exists byte array, and creates new one instead. So, is there any workaround about it?

Answer 1

If you're only interested in writing the messages out, you could go really low level and use CodedOutputStream.writeBytes. It's not how I normally prefer to use protocol buffers, but that's a different matter.

Fundamentally, the problem is that byte[] is mutable, and protobuf messages in Java are meant to be immutable... which means you have to take a copy, basically. Even though the builders themselves are mutable, bytes and string fields are seen as atomic, effectively. (So you can change a builder to have a different byte string or string value for a field, but not mutate the byte string or string itself.)

You should absolutely not be using a string field for data that isn't text though. You should use a bytes field for arbitrary binary data. A string field is expected to contain UTF-8-encoded text.