mDNS does simply not support zone transfers due the way the protocol works. As far as I can tell there are two possible approaches:
1) Try brute force approach, by querying the target (server/subnet). You can do this with dig, just send the query to multicast address and query for your target, eg.
dig -x 192.168.0.10 -p 5353 @224.0.0.251
There are also a few ready scripts and tools that assist in enumerating mDNS targets. Some examples include
- MDNSRecon
- avahi-browse
- mdns-scan
- Bonjour Dumper
- dns-sd -Z and friends
2) Force the daemon to dump its zone file (or settings). You already found out that Avahi obeys
killall -USR1 avahi-daemon
Apple's Bonjour includes mDNSResponder which does not implement dumping zone information. However you can add more logging for similar benefits
A SIGUSR1 signal toggles additional logging, with Warning and Notice enabled by default:
% sudo killall -USR1 mDNSResponder
Once this logging is enabled, users can additionally use syslog(1) to change the log filter for the process. For example, to enable log levels Emergency - Debug:
% sudo syslog -c mDNSResponder -d
A SIGUSR2 signal toggles packet logging:
% sudo killall -USR2 mDNSResponder
A SIGINFO signal will dump a snapshot summary of the internal state to /var/log/system.log:
% sudo killall -INFO mDNSResponder
Also, Wireshark might be used to debug protocol errors. This should be enough for solving interoperability errors.