Stream protobuf bytes to file (python)

Question

I'm writing a python client of a server that has as one of its messages some metadata followed by some required bytes. For my use case, I don't need to have all of the bytes in memory at once, so ideally I'd like to be able to parse the metadata then be able to interact with bytes in a streaming fashion, without them all being pulled into memory first. Is this possible with the python API?

Answer 1

No, the Python API does not support this. In theory, you might be able to accomplish it with a hand-written parser, but it's not guaranteed -- the message writer is allowed to write the fields in any order, so it could very well have placed the bytes first. (In practice, most implementations will write in order of field number -- but that means if you ever add a new field, it's likely to appear after the bytes!)

In general, protobufs are not designed to represent a large amount of data in a single message. See this discussion in the docs for more.