how to dump http traffic?

Question

I am working with webservices, and I need to get a dump of all the HTTP requests and responses, so that I can debug the interoperability between the devices.

I have a small pc with 3 nics that are bridged, so that it acts as an hub and I can tap the traffic. I am looking for a way to easily dump the HTTP traffic, so that I can analyze the SOAP messages exchanged by the two devices.

For I would prefer to implement that with python, I tried scapy with the HTTP extension, but it does not seem to work, because I see the request parsed three times (I wonder if this is due to the used of a bridge) and I am not able to see the responses.

Is there any other way to implement such a tool? I prefer python, but it is not mandatory.

Another small question

I add a subquestion: by using the HTTP interpreter that I linked in the previous question, I see that I sometimes get packets that are only recognized as HTTP and not as HTTPRequest or HTTPResponse. Such packets look gzipped, and I think they are related to the fact that a response body does not fit in a single packet. It there a way with scapy to have these packets merged together? I need a way to get the body of the messages. Again, not only in python, and not only with scapy.

Answer 1

I finally solved my problem by using tshark in pipe and by parsing its output with a python script. Most of the decoding activity is performed by the following command

tshark -l -f "tcp port 80" -R "http.request or http.response " -i br0 -V

which outputs the decoded HTTP packets. and my script performs all the necessary operations.

Answer 2

For the raw sniffing I'd go with tcpdump writing to a pcap file.

tcpdump -i <interface> -s 65535 -w file.pcap  port 80

The -s says write the whole packet out and -w is save. I'm assuming your http goes over port 80, but you can make an arbitrarily complex filter expression. Make sure the interface is the one that leads to the server so you see what it's sending and receiving vs how traffic gets to your bridge host.

You can then parse the pcap with scapy at your leisure, knowing that the capture is happening in a well tested, fast, and reliable manner.

rdpcap("/tmp/file.pcap")

Answer 3

There are some respectable traffic sniffers around already, so you probably have no need to implement one of your own. Wireshark is amongst the most popular. Not only it allows you to capture traffic, but also has some great tools for filtering and analyzing the packets.

sharktools allows you to use Wireshark packet dissection engine from Python, e.g. to filter the packets.

If you have very specific needs or just want to learn something new, pylibpcap is a Python interface for libpcap library, which is used by (almost) every traffic capture program out there.

UPD: Fixed typo in URL for pylibpcap.