How does the Windows kernel debugger in Visual Studio work?

Question

I very recently started attempting to develop a file system driver for Windows, and have started looking at the kernel debugger provided in Visual Studio 2013. The kernel developer is an amazing piece of software -- it is able to debug a live kernel (breakpoints, stack trace, memory access, and everything) of a remote machine connected via an RS232 connection. The one thing I don't understand though -- is how is such debugging possible over a RS232 port? My understanding of the kernel is that it is the piece of software that directly addresses hardware components, and is one of the lowest-level pieces of software on the system. How could something run "above" the kernel, which would allow for the kernel itself to be debugged remotely? How is it that the kernel can be debugged purely in software; without interfering with the software responding to the debug requests on the machine itself (which would be presumably running under the kernel)?

Answer 1

The kernel debugger is neither above nor below the kernel. Rather, it is part of the kernel. Even while you're debugging, and have the kernel stopped, the parts of the kernel that handle the debugger connection are still active.

You will see a number of symbols in the kernel prefixed with Kd, like KdEnableDebugger, for example. These are functions related to Kernel debugging. Kdp functions are private.

Interface-specific parts of the debugger are implemented in kd1394.dll (Firewire), kdcom.dll (Serial), or kdusb.dll (USB). The kernel links against these DLLs to implement the low-level protocol, with functions such as KdSendPacket.

You should check out this very informative article: Kernel and remote debuggers. An excerpt:

The Win2k kernel debugging support is built right into the kernel. The Win2k OS defines a set of routines that cooperatively provide the kernel debugging support to a remote debugger such as WinDbg. Collectively, these routines implement the Win2k OS component called "the kernel debugger".

The basic operation of the kernel debugger is moderately simple. When the target system is normally running, the kernel debugger is quietly sleeping; it's only when certain events occur that it is brought into action. Specifically, it is activated whenever: an exception is raised (either by the processor or by calling ZwRaiseException), a breakpoint is hit, or a native debugging service is requested. Moreover, when the target detects a break-in request from the remote debugger, the kernel debugger is also activated. The kernel debugger's job is to construct a description of the event that occurred in the system and forward it to the remote debugger for the user to analyse.