I’m writing several small operating systems for fun (as you do), and to test and debug them I use the QEMU virtual machine. I’m writing this down to collect the bits and pieces of information I need in one place, and so that I can look it up again if I forget, and hopefully will be of value to others too, for debugging a kernel led me to use features of QEMU and GDB , some of which I was hitherto unacquainted with.
Before we look at QEMU, we should note that a good way to start figuring out what’s going in our kernel is to just disassemble it with
objdump. It’ll show you the disassembled instructions, functions, and addresses of all the things.
objdump -D kernelname.bin
Peering At Interrupts With QEMU
Want to see what interrupts are firing and what the system state is at each point? Use the
-d int option, a sparsely documented feature in QEMU that lets you find out which interrupt got called, what the error code is (if any), and do a register dump upon each interrupt. It’s useful to use along with
-no-reboot so that it doesn’t restart the VM when a triple fault happens.
qemu-system-i386 -d int -no-reboot -no-shutdown -kernel kernelname.bin
A typical interrupt event will look something like this:
0: v=20 e=0000 i=1 cpl=0 IP=0008:0010006f pc=0010006f SP=0010:00107040 env->regs[R_EAX]=00000000 EAX=00000000 EBX=00009500 ECX=00000f9e EDX=00101023 ESI=00000000 EDI=00109000 EBP=00107048 ESP=00107040 EIP=0010006f EFL=00000202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0010 00000000 ffffffff 00cf9300 DPL=0 DS [-WA] CS =0008 00000000 ffffffff 00cf9a00 DPL=0 CS32 [-R-] SS =0010 00000000 ffffffff 00cf9300 DPL=0 DS [-WA] DS =0010 00000000 ffffffff 00cf9300 DPL=0 DS [-WA] FS =0010 00000000 ffffffff 00cf9300 DPL=0 DS [-WA] GS =0010 00000000 ffffffff 00cf9300 DPL=0 DS [-WA] LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy GDT= 00102000 00000017 IDT= 00102020 000007ff CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000 DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 DR6=ffff0ff0 DR7=00000400 CCS=00000010 CCD=00107040 CCO=ADDL EFER=0000000000000000
The first line shows an index number, followed by v (interrupt vector number in hex), e (error number), i (number of times interrupts have fired), IP (the address of the instruction pointer), pc (program counter), SP (stack pointer address), and the last field there is possibly the contents of EAX (told you it was sparsely documented). Following that is a dump of the contents of the registers.
The example above is for 32-bit x86, and it will naturally look different if you’re simulating a different architecture.
Using The QEMU Monitor To Inspect Memory
The QEMU monitor is a console where you can poke at the VM in various ways. You start it with the
-monitor stdio option:
qemu-system-i386 -monitor stdio -kernel kernelname.bin
That tells QEMU to use
stdio (ergo, the terminal) to run the monitor, letting us type in commands and getting a result in our shell window. We frequently need to inspect memory, so to do that we can use
xp. What’s the difference? Using
x will display the contents of a virtual address, while
xp will dump the contents of the physical address. Since it’s early on in my kernel development journey, I don’t have virtual memory yet so here’s an example of a physical memory dump at address
80wx means we want 80 words (the
w part) in hex (the
x part). A word is 32 bits, and other options include
g (64 bits),
h (16 bits), and
b (8 bits). The last bit can be an address or it can be a register that holds an address.
(qemu) xp/80wx 0x106fe4 0000000000106fe4: 0x00000010 0x00000010 0x00000010 0x00000010 0000000000106ff4: 0x00109000 0x00000000 0x00107048 0x00107014 0000000000107004: 0x00107020 0x00000300 0x000b8fa0 0x00000000 0000000000107014: 0x00000020 0x00000000 0x00000000 0x00009500 0000000000107024: 0x00100059 0x00000008 0x00000246 0x001009f4 0000000000107034: 0x00000000 0x00107048 0x00100041 0x00000000 0000000000107044: 0x00000000 0x00000000 0x00100017 0x00009500 0000000000107054: 0x00000000 0x00000000 0x00000000 0x215b8540 0000000000107064: 0x00007f2a 0x0000037e 0x00000000 0x00000000 0000000000107074: 0x00000000 0x215b8840 0x00007f2a 0x00000000 0000000000107084: 0x00000000 0x00000000 0x00000000 0x215b8540 0000000000107094: 0x00007f2a 0x0000035c 0x00000000 0x00000000 00000000001070a4: 0x00000000 0x215b8540 0x00007f2a 0x000003a2 00000000001070b4: 0x00000000 0x00000000 0x00000000 0x00000000 00000000001070c4: 0x00000000 0x00000151 0x00000000 0x2b42ecb0 00000000001070d4: 0x00005586 0x2b4389a0 0x00005586 0x00000000 00000000001070e4: 0x00000000 0x215b8840 0x00007f2a 0x00000000 00000000001070f4: 0x00000000 0x00000000 0x00000000 0x215b8600 0000000000107104: 0x00007f2a 0x2afc360e 0x00005586 0x00000000 0000000000107114: 0x00000000 0x215b83c0 0x00007f2a 0x00000040
You can optionally attach GDB to a QEMU session to get even more help debugging your kernel. You first run QEMU with the
-s option that opens a local port at 1234 to attach your debugger to, and also with the
-S option so it pauses upon startup (type
c in your debugger to continue).
To startup QEMU:
qemu-system-i386 -s -S kernelname.bin
To attach GDB, start it with this (I’m writing my kernel for 32-bit x86 which is why I have a cross-GDB called
And once GDB starts, type:
(gdb) target remote localhost:1234
At this point you can do stuff like set breakpoints before you start the kernel with
c. To set a breakpoint at line 42 in main.c, do this:
(gdb) break main.c:42
There are other ways to pause execution of your code, like an infinite loop. A trick like that is handy when you’re trying to debug hand-written assembly (it’s kernel programming, sometimes you need to). If you want to continue past the infinite loop, you can use gdb to skip to the address of the next instruction (which you can find out via objdump, which let’s say, in this example, is
0x6578). First press
Ctrl-C to pause execution, then type:
(gdb) set $pc = 0x6578
This will set the program counter to
0x6578, then type
c. Presto, you diverted execution to the instruction of your choice.
Finally, a good trick is setting a watchpoint on certain memory addresses that lets you figure out which line of code is causing it to be written to (say you’re getting a stack corruption and you want to watch the stack to see when it’s happening). Let’s say the stack pointer is at
0x107040, do this:
(gdb) watch *0x107040
This will watch for any change to that address. You may need to use it in conjunction with breakpoints to isolate the watchpoint to the section of code that you’re interested in.
This is by no means an exhaustive list of tricks, but I learned these while writing my kernel. These were invaluable to me while I was trying to fish out a bug in my interrupt handler which was a mix of assembly and C, and the assembly part was confusing the debugger. I really needed to be creative when debugging, hence I put infinite loops inside my assembly code, and used the debugger to just jump over them while using QEMU to inspect machine state.