What's the best way to serialize data in a language-independent binary format?

Question

I'm looking into a mechanism for serialize data to be passed over a socket or shared-memory in a language-independent mechanism. I'm reluctant to use XML since this data is going to be very structured, and encoding/decoding speed is vital. Having a good C API that's liberally licensed is important, but ideally there should be support for a ton of other languages. I've looked at google's protocol buffers and ASN.1. Am I on the right track? Is there something better? Should I just implement my own packed structure and not look for some standard?

Answer 1

Given your requirements, I would go with Google Protocol Buffers. It sounds like it's ideally suited to your application.

Answer 2

You could consider XDR. It has an RFC. I've used it and never had any performance problems with it. It was used in ONC RPC and has an and comes with a tool called rpcgen. It is also easy to create a generator yourself when you just want to serialize data (which is what I ended up doing for portability reasons, took me half a day). There is an open source C implementation, but it can already be in a system library, so you wouldn't need the sources.

ASN.1 always seemed a bit baroque to me, but depending on your actual needs might be more appropriate, since there are some limitations to XDR.

Answer 3

Just wanted to throw in ASN.1 into this mix. ASN.1 is a format standard, but there's libraries for most languages, and the C interface via asn1c is much cleaner than the C interface for protocol buffers.

Answer 4

JSON is really my favorite for this kind of stuff. I have no prior experience with binary stuff in it though. Please post your results if you are planning on using JSON!

Answer 5

Thrift is a binary format created by Facebook. Here's a comparison with google protocol buffers.

Answer 6

Check out Hessian

Answer 7

There is also Binary XML but it seems not stabilized yet. The article I link to gives a bunch of links which might be of interest.

Answer 8

Another option is SNAC/TLV which is used by AOL in it's Oscar/AIM protocol.

Answer 9

Also check out Muscle. While it does quite a bit, it serializes to a binary format.

Answer 10

Few Thing's you need to Consider

1. Storage
2. Encoding Style (1 byte 2 byte)
3. TLV standards

ASN.1 Parser is the good for binary represenations the best part is ASN.1 is a well-established technology that is widely used both within ITU-T and outside of it. The notation is supported by a number of software vendors.