HTTP requests trace

Question

Are there any tools to trace the exact HTTP requests sent by a program?

I have an application which works as a client to a website and facilitates certain tasks (particularly it's a bot which makes automatic offers in a social lending webstite, based on some predefined criteria), and I'm interested in monitoring the actual HTTP requests which it makes.

Any tutorials on the topic?

Answer 1

Some popular protocol/network sniffers are:

• Wireshark (previous the famous Ethereal)
• Nirsoft SmartSniff (using WinPcap)
• Nirsoft SocketSniff (allows you to watch the WinSock activity of the selected process and watch the content of each send or receive call, in Ascii mode or as Hex Dump)
• Microsoft's Network Monitor (and a list of video-tutorials here, note video 'Advanced Filtering 2 of 2' where they specifically filter on process)

Wikipedia article 'Comparison of packet analyzers' has a nice overview of some other tools to.

Alternatively you could also look into (man-in-the-middle) proxy tools like:

• Fiddler
• mitmproxy

Both of the above actually record/decrypt/modify/replay HTTPS to!! You'd need to point the application you are monitoring to this proxy. If nothing else uses that proxy the log would be application/process specific and another upside to this approach is that one could also run the monitor/logger on a different machine.

Once you choose a tool, you can easily google a tutorial to go along with it.
However the core idea is usually the same: basically one sets a filter (on capture itself or display of captured data) on things like protocol, network/mac address, portno, etc. Depending on the tool, some can also filter on local application.

Hope this helps!

Answer 2

Take a look at HTTP Toolkit (disclaimer: it's my project).

Totally automatic HTTP & HTTPS interception, with zero setup, isolated to just the code you want to debug.

You can open a browser with it, and see all the traffic from that one window immediately (but no others), or run a terminal and automatically see all traffic only from processes started from that terminal. Built-in HTTPS decryption for everything, with no risky system-wide certificates and no manual setup. Let me know what you think!