It's kind of impossible to answer your question adequately, because it depends on your specific requirements and needs.
To some extend you could say it's the amount of data used to describe the serialized representation, which influences the performance.
For example, a XML document would need opening and closing tags like ..., while JSON would suffice with prop1:"...", which reduces the amount of character data needed to describe the instance. Binary might even skip the property naming, which then further reduces the amount of data needed to describe the serialized instance.
Although this might paint the picture binary serialization would be the fastest, it isn't always the case. It depends on how much time the serializer needs to get from an object instance to the serialized description.
If you really need performance, then you should take a look at the google protocol buffers. In my experience these are lightning fast and quite easy to use.