Debugging A Kernel In A Virtual Machine

I’m writing several small operating systems for fun (as you do), and to test and debug them I use the QEMU virtual machine. I’m writing this down to collect the bits and pieces of information I need in one place, and so that I can look it up again if I forget, and hopefully will be of value to others too, for debugging a kernel led me to use features of QEMU and GDB , some of which I was hitherto unacquainted with.

Before we look at QEMU, we should note that a good way to start figuring out what’s going in our kernel is to just disassemble it with objdump. It’ll show you the disassembled instructions, functions, and addresses of all the things.

objdump -D kernelname.bin

Peering At Interrupts With QEMU

Want to see what interrupts are firing and what the system state is at each point? Use the -d int option, a sparsely documented feature in QEMU that lets you find out which interrupt got called, what the error code is (if any), and do a register dump upon each interrupt. It’s useful to use along with -no-shutdown and -no-reboot so that it doesn’t restart the VM when a triple fault happens.

qemu-system-i386 -d int -no-reboot -no-shutdown -kernel kernelname.bin 

A typical interrupt event will look something like this:

     0: v=20 e=0000 i=1 cpl=0 IP=0008:0010006f pc=0010006f SP=0010:00107040 env->regs[R_EAX]=00000000
EAX=00000000 EBX=00009500 ECX=00000f9e EDX=00101023
ESI=00000000 EDI=00109000 EBP=00107048 ESP=00107040
EIP=0010006f EFL=00000202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00cf9a00 DPL=0 CS32 [-R-]
SS =0010 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     00102000 00000017
IDT=     00102020 000007ff
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
CCS=00000010 CCD=00107040 CCO=ADDL    
EFER=0000000000000000

The first line shows an index number, followed by v (interrupt vector number in hex), e (error number), i (number of times interrupts have fired), IP (the address of the instruction pointer), pc (program counter), SP (stack pointer address), and the last field there is possibly the contents of EAX (told you it was sparsely documented). Following that is a dump of the contents of the registers.

The example above is for 32-bit x86, and it will naturally look different if you’re simulating a different architecture.

Using The QEMU Monitor To Inspect Memory

The QEMU monitor is a console where you can poke at the VM in various ways. You start it with the -monitor stdio option:

qemu-system-i386 -monitor stdio -kernel kernelname.bin

That tells QEMU to use stdio (ergo, the terminal) to run the monitor, letting us type in commands and getting a result in our shell window. We frequently need to inspect memory, so to do that we can use x or xp. What’s the difference? Using x will display the contents of a virtual address, while xp will dump the contents of the physical address. Since it’s early on in my kernel development journey, I don’t have virtual memory yet so here’s an example of a physical memory dump at address 0x106fe4. The 80wx means we want 80 words (the w part) in hex (the x part). A word is 32 bits, and other options include g (64 bits), h (16 bits), and b (8 bits). The last bit can be an address or it can be a register that holds an address.

(qemu) xp/80wx 0x106fe4
0000000000106fe4: 0x00000010 0x00000010 0x00000010 0x00000010
0000000000106ff4: 0x00109000 0x00000000 0x00107048 0x00107014
0000000000107004: 0x00107020 0x00000300 0x000b8fa0 0x00000000
0000000000107014: 0x00000020 0x00000000 0x00000000 0x00009500
0000000000107024: 0x00100059 0x00000008 0x00000246 0x001009f4
0000000000107034: 0x00000000 0x00107048 0x00100041 0x00000000
0000000000107044: 0x00000000 0x00000000 0x00100017 0x00009500
0000000000107054: 0x00000000 0x00000000 0x00000000 0x215b8540
0000000000107064: 0x00007f2a 0x0000037e 0x00000000 0x00000000
0000000000107074: 0x00000000 0x215b8840 0x00007f2a 0x00000000
0000000000107084: 0x00000000 0x00000000 0x00000000 0x215b8540
0000000000107094: 0x00007f2a 0x0000035c 0x00000000 0x00000000
00000000001070a4: 0x00000000 0x215b8540 0x00007f2a 0x000003a2
00000000001070b4: 0x00000000 0x00000000 0x00000000 0x00000000
00000000001070c4: 0x00000000 0x00000151 0x00000000 0x2b42ecb0
00000000001070d4: 0x00005586 0x2b4389a0 0x00005586 0x00000000
00000000001070e4: 0x00000000 0x215b8840 0x00007f2a 0x00000000
00000000001070f4: 0x00000000 0x00000000 0x00000000 0x215b8600
0000000000107104: 0x00007f2a 0x2afc360e 0x00005586 0x00000000
0000000000107114: 0x00000000 0x215b83c0 0x00007f2a 0x00000040

Using GDB

You can optionally attach GDB to a QEMU session to get even more help debugging your kernel. You first run QEMU with the -s option that opens a local port at 1234 to attach your debugger to, and also with the -S option so it pauses upon startup (type c in your debugger to continue).

To startup QEMU:

qemu-system-i386 -s -S kernelname.bin

To attach GDB, start it with this (I’m writing my kernel for 32-bit x86 which is why I have a cross-GDB called i686-elf-gdb):

i686-elf-gdb kernelname.bin

And once GDB starts, type:

(gdb) target remote localhost:1234

At this point you can do stuff like set breakpoints before you start the kernel with continue or c. To set a breakpoint at line 42 in main.c, do this:

(gdb) break main.c:42

There are other ways to pause execution of your code, like an infinite loop. A trick like that is handy when you’re trying to debug hand-written assembly (it’s kernel programming, sometimes you need to). If you want to continue past the infinite loop, you can use gdb to skip to the address of the next instruction (which you can find out via objdump, which let’s say, in this example, is 0x6578). First press Ctrl-C to pause execution, then type:

(gdb) set $pc = 0x6578

This will set the program counter to 0x6578, then type continue or c. Presto, you diverted execution to the instruction of your choice.

Finally, a good trick is setting a watchpoint on certain memory addresses that lets you figure out which line of code is causing it to be written to (say you’re getting a stack corruption and you want to watch the stack to see when it’s happening). Let’s say the stack pointer is at 0x107040, do this:

(gdb) watch *0x107040

This will watch for any change to that address. You may need to use it in conjunction with breakpoints to isolate the watchpoint to the section of code that you’re interested in.

Conclusion

This is by no means an exhaustive list of tricks, but I learned these while writing my kernel. These were invaluable to me while I was trying to fish out a bug in my interrupt handler which was a mix of assembly and C, and the assembly part was confusing the debugger. I really needed to be creative when debugging, hence I put infinite loops inside my assembly code, and used the debugger to just jump over them while using QEMU to inspect machine state.