Kernel-debugging a Qemu windows VM from a Linux host using wine

The "normal" way to debug a Qemu/KVM windows virtual machine at kernel-level (e.g., for writing a windows driver) is to use 2 windows VMs. One (called server) in which to run the kernel you are going to debug and one (called client or host) in which to run the debug tools.

(If you want to try this way, the instructions here have worked for me at least).

Several people have wondered if it is possible to save (at least some of) the memory overhead of the host/client VM by using wine instead to run the debug tools (from the Linux host). Unfortunately this seems to be harder than it seems.

The problem is that the client-side windows debugger (dbgeng.dll) supports only serial, firewire or windows named pipe transports for the windows debug protocol (KDCOM). Qemu/KVM can output its virtual serial port data to a real serial port, a file, a unix domain socket or a TCP socket on the host.

The first thought was to see if a pty would do as the serial port, but an experiment with socat indicated that this can't work. There also isn't any facility in wine for interoperability between windows named pipes and Linux domain sockets, so a "real" serial port seems to be the only option for dbgeng.dll and qemu to talk through.

Unfortunately there doesn't seem to be a free software "virtual serial port"/"virtual null modem" solution for linux like com0com for windows (an interesting introductory project if someone is interested, these days such virtual serial port can even be implemented in userspace using cuse).

But luckily, there exists a "free as in beer" solution, named VSPDL, that consists of a kernel (GPL) and a userspace (closed-source) component. So, with KVM, a windows XP VM, wine, debugging tools for windows and VSPDL we can now assemble a (somewhat) workable solution:

Instructions

1. download debug tools for windows (preferably 6.10 since 6.12 and above are not available standalone anymore) and set them up in wine. I used wine 1.3.17. You should be able to launch kd with wineconsole and windbg with wine without problems, although they won't do much for now.

2. Setup VSPDL (you can use these instructions for the most part. The only thing I had to change for ubuntu maverick / vspd-1.44 is in lib/src/sysheader.h from linux/autoconf.h to generated/autoconf.h and make a symlink from linux-headers-`uname -r` to linux-`uname -r`). In case it is useful, my configuration file for vspd is available here. At this stage, it would be good to use something like minicom or cutecom to verify that the 2 virtual serial ports created from the config file (/dev/vsps0 and /dev/vsps1) can exchange data fine.

3. Add "-serial /dev/vsps0" to the script that launches your VM.

4. Do the guest settings described here to enable debugging in windows kernel.

5.

ln -s /dev/vsps1 ~/.wine/dosdevices/com1

and

sudo chmod 666 ~/.wine/dosdevices/com1

or use a more secure (distro-dependent) way to get write access to the device, like adding your user to the appropriate group or something along the lines of ConsoleKit configuration.

6. launch the VM

7. launch kd.exe with

WINEDLLOVERRIDES="dbgeng,dbghelp=n" wineconsole ~/.wine/drive_c/Program\ Files/Debugging\ Tools\ for\ Windows\ \(x86\)/kd.exe -b -k com:port=com1,baud=115200

And you should be able to see something resembling the screenshot at the top of the article :)

Caveats: Unfortunately this system is still not fast, reliable or stable enough for a usable debugging session. E.g., the virtual serial port has the problem that it pretty much requires a vmexit for each byte that is being transferred through it. Even if we don't consider this we have this situation of having to do "back and forth" through the kernel (guest->kernel->host->kernel virtual serial port->vspd->kernel other virtual serial port->wine) all the time to get the information back and forth.

Not to mention the constraint that when reading memory, the kdcom protocol imposes a limit of 4k chunks of memory per packet, which means that memory dumps can be reeeeeeeeeeally slow.

Long term, better solutions are needed for this problem of Kernel debugging windows VMs from wine, but at least we have seen that it is possible.

Pantelis

UPDATE: by default VSPDL's vspm module produces a huge amounts of log information during a debug session (gigabytes). This information seems useless and it can even overflow your /var, so it is a good idea to either filter it somehow (rrd?) or just set DEBUG=0 instead of 1 in vspm.conf