How can I reverse engineer an application's protocol?

Question

I'm using an application (an instant messenger) which is not very popular. I'm trying to find the protocol that it uses. I know it's using TCP/IP but I want to find out all the commands that it is sending to the server and receiving from the server.

I tried a couple of sniffers, but they can not recognize this application by name and more over all I got was some unrelated hexadecimal codes.

Is there any idea how I can find the application's specifications?

(Please note: I googled it and found nothing and also there is no documentation by the author.)

Answer 1

There are generally two approaches to reverse engineering something like this:

You could try disassembling it with a tool like IDA PRO.

You could try sniffing its traffic with a tool like Wireshark

Either way, it's likely to be a LOT of work.

Answer 2

Wireshark will tell you the protocol. The fact that you cannot read the messages in clear text on the wire is a good thing, isn't it?