Reconstructing data from PCAP sniff

Question

I am trying to sniff HTTP data through libpcap and get all the http contents (header+payload) after processing the TCP payload.

As per my discussion at Writing an http sniffer (or any other application level sniffer) , I am facing problems due to fragmentation - I need to reconstruct the whole stream (or defragment it) to get a complete HTTP packet, and this is where I need some help.

Thanks in anticipation !!

Answer 1

It's really pretty simple. Just take the ethernet frames that you get from pcap and extract the IP packets from them, reassembling any that were fragmented. Then, reorder the TCP segments from the IP packets, according to the sequence numbers, paying attention that you discard any duplicate data. Then, process the stream as an HTTP stream. Of course, HTTP doesn't come in packets; it is an application layer protocol, but I'm sure this will be obvious once you've done all this other work. Pay attention as you do all these things to checksum the IP headers and TCP segments, to ensure that your data is correct. Also, if pcap happens to miss any packets, make sure you deal with this appropriately.

To help you along the Linux TCP stack should provide a concise reference to this process as it occurs in the kernel.

Answer 2

Rather than reassemble the streams youself, you can use tcptrace to reassemble the pcap file. I believe -e will do it.

Once you have the application-layer data in one piece, you can apply simple HTTP header parsing.... Perhps from a library such as http://github.com/ry/http-parser

Answer 3

To reconstruct the data contained in a pcap file a wonderful tool is Xplico: http://www.xplico.org

Answer 4

The best tool to recostruct http content from pcap files is justniffer. It uses prtion of linux kernel for IP fragmentation e tcp packet reordeiring.

Answer 5

PCapPlusPlus includes an example console program TCPReassembly which sniffs traffic and outputs each stream to a separate text file. You can instruct which stream to listen to, amongst many options.

The documentation also mentions a linux app tcpflow with even more options.