All posts by JohanJ

oh boy

In February I had been working on Linux (Ubunut) for a couple of years without paying too much attention to the system. However I had to reinstall Ubuntu on a new laptop through the corporate firewall and VPN, plus make all the special built company tools to work.

It was a nightmare. The combination of being blocked by the firewall, not having access on the VPN and missing drivers and libraries for building the tools forced me to really learn about the OS – the hard way. I got so tired of reinstalling that I decided to cheat and use VM (virtualbox) instead. I did not get repelled by Linux, instead I got really fascinated by the flexibility and community around the OS. I have changed distro to Arch, that was the weirdest thing I ever seen when I first tried it out. I thought I couldn’t get more geeky.

Until today, when I realized that I am installing debian on a mips processor through qemu on my arch, so that I can easily objdump mip elf files (because my router is a dlink and I am curious how it works).




Always relevant:


virtualbox – OS development and debugging

The Debugger for virtualbox is somewhat unknown. There are not a lot of resources online about it and it is not even visible by default. However it is a really powerful tool that makes the sensitive and low level implementation of OS much easier.

See my earlier post how to enable the debugger on a windows host.


I have followed the code in the little book of OS development and reached chapter 5 about segmentation. Segmentation is a low level task that is best suited for pure assembler which makes it very difficult to know if you did anything wrong. Or more precisely, it is difficult to know what you did wrong, since an error will generate a crash. Logging or printing to the screen will often not help you.

The debugger gives you a tool to observe your system from the outside, relatively immune from the crash and with plenty of command to show low level info such as registers and address locations.

Writing “help commands” will give you a huge list of debug commands. You can see more info about the separate commands by typing “help <command>”

Finding points of interests

The first thing to do is to find where in memory interesting code is. Looking for hex pattern is quite difficult. A better option is to look for strings used by the OS. For an example, I have the string “Good bye sir” once the kmain function is done, before the infinity jmp loop (see the OS Development).

Searching for a string can be made by sa <range> <pattern>.

If you have a string, let’s say “Good bye sir” you can search for where it is in memory with:

>sa 0 “Good bye”

The output should be something like:

VBoxDbg> sa 0 “Good bye”
%%000000000010054a: 47 6f 6f 64 20 62 79 65-20 73 69 72 2e 0a 00 00 Good bye sir….

Next we would like to see what else there is in the memory next to the “Good bye string”. Displaying byte info about a memory area is done by db <address>

Usually your code is in the nearby of strings if you have made a small assembler program for booting (but it is certainly not true for larger programs).

VBoxDbg> db 10054a
%000000000010054a: 47 6f 6f 64 20 62 79 65-20 73 69 72 2e 0a 00 00 Good bye sir….
%000000000010055a: 00 00 02 b0 ad 1b 00 00-00 00 fe 4f 52 e4 b8 be ………..OR…
%000000000010056a: ba fe ca bc 04 18 10 00-e8 89 fa ff ff eb fe 00 …………….
%000000000010057a: 00 00 14 00 00 00 00 00-00 00 01 7a 52 00 01 7c ………..zR..|
%000000000010058a: 08 01 1b 0c 04 04 88 01-00 00 1c 00 00 00 1c 00 …………….
%000000000010059a: 00 00 64 fa ff ff 64 00-00 00 00 41 0e 08 85 02 ..d…d….A….

There are no more strings after that line. Then the data below might be executable instructions. You can show what the hex is in assemble by  u32 <address>

If we take a look at the memory address we got above:

VBoxDbg> u32 10055a
%000000000010055a 00 00 add byte [eax], al
%000000000010055c 02 b0 ad 1b 00 00 add dh, byte [eax+000001badh]
%0000000000100562 00 00 add byte [eax], al
%0000000000100564 fe 4f 52 dec byte [edi+052h]
%0000000000100567 e4 b8 in AL, 0b8h
%0000000000100569 be ba fe ca bc mov esi, 0bccafebah
%000000000010056e 04 18 add AL, 018h
%0000000000100570 10 00 adc byte [eax], al
%0000000000100572 e8 89 fa ff ff call 000100000h
%0000000000100577 eb fe jmp -002h (000100577h)

As you can see, there is the “cafebabe”, a call to where we placed our kernel (“call 000100000h”) and the endless jump loop in the end “jmp -002h”.

We want to pause the program so we need to add a BP. There are several options for this. I use br <address>

I’m not sure about the  the difference between bp and br. However my system crashed if I try to use bp. I suggest that you try both.

>br 100577

If you want to see all your BP use bl

VBoxDbg> bl
0x4 e 1 r 0000000000100577 0000 (0000 to ~0)

Now we have paused our program. If we would like to execute the next command/trace we can do that by t

VBoxDbg> t
dbgf event: Single step! (rem)
eax=00000000 ebx=0002cd80 ecx=0000000d edx=0000001b esi=0002cef0 edi=0002cef1
eip=00100577 esp=00101804 ebp=00067ee0 iopl=0 nv up di pl zr na po nc
cs=0008 ds=0010 es=0010 fs=0010 gs=0010 ss=0010 eflags=00000046
0008:00100577 eb fe jmp -002h (000100577h)

Perhaps not a very useful BP since it is in the jmp loop.

There are plenty of nice dump commands, example to check your GDT like you set up in chapter 5 you can use dg

VBoxDbg> dg
0008 CodeEO Bas=00000000 Lim=fffff000 DPL=0 P NA G BIG AVL=0 L=0
0010 DataRW Bas=00000000 Lim=fffff000 DPL=0 P NA G BIG AVL=0 L=0


A problem when debugging is that you often need to set a BP before  the OS boots up, because the OS won’t halt and you will miss the code execution point. For a simple OS you can often reuse the same memory address, like 0x100577, and if you followed the OS book then 0x100000 will be the C code entry. The VM will start in a paused state, so add the BP and then resume.

virtualbox – The Little Book About OS Development

Since I want to learn more about Linux  Device Driver development I need to learn more how the OS work from scratch. There are many tutorials about the Linux kernel, but they are often too heavy. I need a guide that show me in small steps how to set things up. There is a one great tutorial:  The little book about OS development. I am really happy that the authors put up all the effort to write this and publish it for free! It is not Linux but it is still based on the same problems that most OS has to deal with during boot up.

Note that it is a bit “old school”, some techniques are not well suited for developing a real OS, how ever if you like me want to get a better understanding of multitasking, virtual memory IRQ and such it is perfect. IF you plan to make a real OS you should read beginners mistakes at the Osdev wiki, quite a lot of these faults are in the little book. Another thing to keep in mind is that most links are outdated, I found this tutorial to be very well written (down to explaining the chip architecture!)

I am a big fan of virtualbox so I decided to update the tutorial with how to use it instead of the brosh emulator that is used in the book.

You could either develop directly on a Linux host. This will save you time. However I’m on a windows host and I will use a Linux Guest to develop on. If you use Linux as a host you can skip the part about exporting the iso-file.


Let’s follow until the first step of the book: hello CAFEBABE

We need to create a guest OS that will run the compiled OS. Let’s call it guest C (as Compiled or CAFEBABE). You can optionally develop from a guest Linux OS. If you are on a host Linux this is not needed. Let’s call this OS guest L.

First create guest C:

  • No EFI ( let’s not make it complicated)
  • Don’t create any HD. We will boot from the ISO.

Optional. Set up Guest L:

  • Add a shared folder and name it osdev. Enable auto mount.
  • In guest L, add your user to vboxfs group: $adduser $USER vboxfs. Reboot.

Let’s make an export script*. Create a file create_iso. Add the gensioimage command and copy the iso to the shared folder:

genisoimage -R -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -A os -input-charset utf8 -quiet -boot-info-table -o os.iso iso
#copy iso to shared folder
cp os.iso /media/sf_osdev

This is a very simple script. You should probably add some error handling if an iso couldn’t be created.

*In later chapters we will use a make file. I suggest to add the cp line in that make file:

os.iso: kernel.elf
cp kernel.elf iso/boot/kernel.elf
genisoimage -R -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -A os -input-charset utf8 -quiet -boot-info-table -o os.iso iso
cp os.iso /media/sf_osdev/

Running the guest and debugging

Once you got the iso (if you exported from guest L it will be in osdev folder on your host) you can add it to an optic drive on your C guest. Running it is not very fun, since you can’t see any registers – we need to enable debugging.

By default debugging is disabled, see the manual. There are a couple of options to enable it. For Linux it is quite easy since it is a command line friendly OS. If you use Windows as host the easiest will be the following option (quoted from the manual):

“Set the VBOX_GUI_DBG_ENABLED or VBOX_GUI_DBG_AUTO_SHOW environment variable to true before launching the VirtualBox process. Setting these variables (only their presence is checked) is effective even when the first VirtualBox process is the VM selector window. VMs subsequently launched from the selector will have the debugger enabled.”

In Windows this is done by opening a command promt and entering the following:


Restart virtualbox and start guest C.

It is paused by default, so press ctrl  p. You will see info that the kernel is loaded. To see the 32 bit registers use the debug command: rg32:



As you can see there’s a CAFEBABE in eax.


Installing Gentoo on EFI and virtualbox is tough!

The problem is because the gentoo CD doesn’t come with a EFI boot, so you have to start from MBR. Well, it turns out that it’s not possible to change the EFI settings if you have booted through MBR.

That means you have to install gentoo without EFI support first and make a boot partition ready for the EFI boot loader. How do you do that without being able to configure EFI?

If you are just looking for the solution go to the short version.

The Fix – Long Version

First of all in Gentoo you must enable EFI stub so that the kernel can be executed as a boot loader (awesome functionality!).

Secondly if EFI has nothing specified it will look for a file named “boot<arch>.efi” in \efi\boot. So in most cases ‘\efi\boot\bootx64amd.efi‘. Notice that since EFI requires FAT the path uses backslash rather than forward slash (/). So you need to copy the kernel there and rename it.

After rebooting (and turning on EFI in virtualbox) the kernel loads. But panics:

VFS: Cannot open root device (null) or unknown-block(0 0): error -6
Please append a correct “root=” boot option; here are the available partitions:
(omitted sda partitions)
Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

What does this mean? At first I thought it was because I didn’t set any EFI variables. It is possible to do so through the EFI Shell. You can start this by mashing F12 when virtualbox is booting up a VM. From here there are many commands you can run to tweak your boot up. And written by some drunk lemurs, really wtf Intel!

Let’s have a look at some useful commands:

ram -r: shows the boot options.

bcfg: adds boot options. Example is bcfg boot add <option nr> <efi file> “<label>” . In my case this command would look like blk3:\efi\boot\bootx64.efi

I never figured out how to add kernel arguments (there’s a “-opt” option to bcfg).

This did not help, I still got kernel panic. My second thought was that it was because I did not have the driver for the HDD. I recompiled the kernel with anything that rhymed with EFI,SATA ,PIIX or ACHI. It didn’t help. The live  CD could find the drives, so I took every module it used and put into /etc/conf.d/modules by lsmod | gawk ‘{print $1}’. It didn’t help (I got quite a lot of warnings though).

Thinking about it, since the kernel actually got loaded it must have found the device. The problem is probably the ‘correct “root=”…’ message.

It turns out that you can set the root partition by a kernel command line option. I had no idea, but this is something boot loaders usually do. I heard you could also add these options when you compile the kernel, but I don’t want to recompile my kernel. Instead I booted up in UEFI Shell again and started the efi file with a root option:

>blk3:\efi\root\bootx64amd.efi root=/dev/sda2

It worked! Gentoo boots, and I got a warm and nice feeling in my stomach. Since it has booted from EFI it should be possible to set the parameters through efibootmgr:

$efibootmgr -c -d /dev/sda -p 1 -L “gentoo Linux” -l ‘\efi\boot\bootx64amd.efi’  root=/dev/sda2
$efibootmngr -v

The output shows it worked. But after reboot still kernel panic. Checking the boot options in UEFI Shell I can see that there’s no record saved. Is it a bug in efibootmgr? I spent a few more hours trying to add options with bcfg when I realized there’s a really nice way to add boot options. Don’t boot the shell, instead enter Boot Maintenance Management. From here it’s very easy to add a record with extra arguments.

Starting is fine, but as soon as I reboot the EFI variables will be wiped! This must be a bug (feature?) in virtualbox.

There is one final resort. If EFI does not find an efi file in ESP it will try to execute startup.nsh in the efi\boot folder. Since it will execute the efi file first it’s important to rename it. I renamed mine to gentoo.efi.

Create a startup.nhs file with the UEFI Shell command:


save it and reboot.

Finally Gentoo can automatically boot up in EFI mode!

The fix – short version

  • First install gentoo and create a FAT32 boot partition (and all other things needed for EFI, such as GPT and ESP). Don’t forget to update fstab.
  • On first reboot enter the boot menu by pressing F12.
  • Go to Boot Maintenance manager -> Boot Options -> add boot options
  • Select the boot partition, probably the only one with a GUID. Find the location of your efi file (efi\boot\bootxamd64.efi)
  • Add description (any thing you like). Input Option Data: root=/dev/sda2
  • Commit changes and exit. Go back and select boot manager and select your created boot option.
  • Once gentoo boots up you can try to add a record from efibootmgr. If it doesn’t save after reboot continue with the steps below.
  • Rename the efi file to something like /efi/boot/gentoo.efi
  • Create startup.nsh in the same folder and add the UEFI Shell path to the efi file. Example: blk3:\efi\boot\gentoo.efi
  • Reboot.