How can you make your Wireshark dissector disregard packets until the beginning of a valid stream is observed?

Question

I have a dissector built in Lua which simply checks if packets begin with a zlib header. From that point, I want to decompress that stream in real-time.

Unfortunately, I am getting stream corruption because only one direction (server to client) is compressed and the client interferes with that. I want one dissector to handle the server to client packets and another dissector to handle the client to server packets, since they use essentially different protocols, or a way to handle them both in one dissector.

The tricky part also is that you cannot decompress the zlib stream without seeing the very first packet in the stream, so I want to display my packets in some other color until the start of the compressed stream is observed, possibly filtering them out.

I tried return 0 and return nil to indicate that I cannot process the packet with the dissector, but it's still tagging it with my dissector's protocol, because the dissector registered itself against the port the packet came across on. This didn't change anything, so it seems.

• How can I tell Wireshark that my dissector did not successfully process a packet?
• How can I defer a packet to my client to server dissector when it receives a packet destined for the port I am registered to?
• How can I get my dissector to ignore packets until it sees the beginning of the zlib stream?

Answer 1

• How can I tell Wireshark that my dissector did not successfully process a packet?

You can't - if you've registered your protocol dissector for a specific TCP or UDP port, then packets for that UDP/TCP port are your protocol. Returning 0, nil, or false won't change that. As of the more recent Wireshark 1.11.3 builds you can register a heuristic dissector instead though. (more on that below)

• How can I defer a packet to my client to server dissector when it receives a packet destined for the port I am registered to?

Have your proto.dissector() function call one of two other functions you define. I.e. write a local function called process_client and another called process_server and call those functions within your proto.dissector() function, based on whatever factors make it one vs. the other. (sounds like the source vs. dest port would be the decider in your case?)

• How can I get my dissector to ignore packets until it sees the beginning of the zlib stream?

It depends on how you mean "ignore". Obviously you need to dissect every packet on that port... at least enough to figure out if it's the beginning one you're looking for. Usually what dissectors (the C-code ones) do in such cases is they mark the packet as a "Continuation" - i.e., it's their protocol, but they set the Info column to say the text "Continuation" for a packet they can't dissect. It's pretty common to see that with TCP-based protocols at the beginning of capture files, because it's pretty common to start capturing in the middle of a TCP stream and not get the whole application message in the first TCP segment(s).

Another option is to use a heuristic dissector, which recently got introduced into 1.11.3. A heuristic dissector is similar to a dissector except the name of the function is different, and it's registered slightly differently. (the API is documented here) The dissector.lua script at the top of the Wireshark Lua examples page implements a heuristic dissector.

A heuristic dissector function gets called for every packet like a normal dissector, but it returns true or false - if it returns false, Wireshark tries any other registered heuristic dissectors and doesn't mark it as being your protocol; if you return true, wireshark knows it's yours. And you can set the "conversation" as being your protocol and then wireshark will call your "normal" dissector function from then on for that source/dest IP:port combination. In other words, you don't register your "normal" dissector, just the heuristic one, and then you tell wireshark when to start using your normal one for that flow.

---

As an aside: generally you'd be better off asking Lua-based Wireshark questions on ask.wireshark.org, as there are more people there with experience in that, and usually you get answers faster.