I have a hard time imagining anything that contains only "6 primitive values" could take 7ms to construct. That's perhaps 100,000 times as long as it should take. So I'm not sure I understand what you're doing.
That said, protobuf builders are indeed more complicated than a typical POJO for a number of reasons. For example, protobuf objects keep track of which fields are currently set. Also, repeated primitives are boxed, which makes them pretty inefficient compared to a Java primitive array. So if you measure construction time alone you may see a significant difference. However, these effects are typically irrelevant compared to the time spent in the rest of your app's code.