Journey to the Core of the Computer!

Alright, it’s been quite a while! With the ‘completion’ of a basic bare-metal programming model, this blog describes my journey in Operating Systems development, from a simple bare-metal program to an almost cross-hosted operating system, all built from scratch! Along the way, I got a glimpse of why operating systems work and why developing an operating system in one of the toughest tasks there is. Working with something so low-level is exciting and enriching. Via this blog, I want to share my journey with the readers and discuss the ups and downs related to such low-level development, and hopefully spur the reader to take an interest in this field. 😄

Getting Started

I’ve always been fascinated by the way things work. “How does it work?” is often the first question that pops into my head when I see something interesting. It was this curiosity that led me to explore the field of Computer Architecture, an area of Computer Engineering that deals with — you guessed it — how computers are designed and how they work. Reading more about this field led me to the area of Operating Systems, a software abstraction layer over your hardware. That is, an OS is low-level software that makes the hardware “look pretty” so that the user can access it with ease. Thus, the operating system is the glue that binds hardware to user-level software, and one needs to know the hardware architecture intimately to be a sound operating systems/embedded systems developer.

Here begins our adventure! Where we construct this low-level software from scratch, learning and exploring hardware architecture and organization as we proceed.

The Bootloader

The bootloader (bootstrap loader) is a piece of software that is responsible for setting up the device and loading our kernel/operating-system. As a programmer, the bootloader is the first place where he gets control. However, bootloader programming is often considered difficult, and many seasoned developers advise using ready-made bootloaders like GRUB so that the programmer may focus on his actual operating system.

However, being a rebel, I decided to make my very own tiny bootloader for KSOS: a lot of assembly language, a lot of work, and a lot of pain. However, it was one of the most rewarding experiences of this project, as I could clearly see and appreciate the effect of every bit of instruction (pun intended 😛) in software. Dr. Nick Blundell’s book helped a lot in making my bootloader.

The bootloader is a pretty sweet one! It’s written purely in x86 assembly and has some really nice features, like a mini parser for a FAT filesystem. The bootloader parses the filesystem to read the kernel from the disk and load it into memory. Although making the FAT filesystem parser was one of the most challenging tasks, in the end, it was all worth it. 😃

However, moving forward, I would like to port over to something ready-made like GRUB: it’s well documented, portable, and very reliable. It’s such a beast that it gives operating systems a run for their money!

Okay, maybe a lot more than 4 times.

The Kernel

Once I had a bootloader that was good enough to read disks and manipulate memory, it was time to start with the kernel. I was immediately fascinated by the interaction between C and assembly code, where one has to really appreciate the meaning of assembly, compilation, linking, and loading, consequently getting all the pieces working together. Once it does, viewing the interplay between all the modules is absolutely fascinating!

To have an understanding of this linking process and C-assembly mixture, I had to spend quite a lot of time analyzing the generated binaries and how they transformed on linking. This was one of the biggest struggles of the project, and to be frank, it’s not that easy to look at binary code without wanting to kill yourself every five minutes. 😭

Testing out this tiny ‘kernel’ was simple. I wrote a few simple routines for writing to the screen (printf-ish C routines). You read it right; I made my own printf statement! I could write messages on the screen, change the colors to something fancy, and a lot of other cool stuff.

Soon, I wrote specialized modules for necessary memory management, interrupt handling, and fundamental device drivers, about which I’ll get back to later!

Build System

Let’s take a quick detour from our operating system, shall we?

In a project like OS development, where one builds everything from scratch, they must maintain a clean codebase with an efficient build system, i.e., some automated way to compile, assemble, link, install, etc. all of their modules. Moreover, this isn’t like application/web development with ‘pretty’ IDEs and extreme framework support. We ARE the framework! One needs to put careful thought and action regarding the management of this build system, since messing up here is very costly!

Therefore, a significant amount of my time was spent in structuring and restructuring my codebase and build system, from simple shell scripts to complicated makefiles for the operating system. Right now, the build system uses a combination of makefiles and a shell script for efficiency and is heavily inspired by the Meaty Skeleton Page on OSDev wiki.

Perhaps something to further explore in the future would be some other third-party open-source build systems like CMake, Autoconf, etc., and to port them to our project.

Kernel Modules

Memory Management

As soon as I had a kernel with some basic screen functions (like puts()), I decided to create a memory management module. In the long run, a kernel must have a memory management unit of some sort. Every process uses memory, and it is the responsibility of the kernel to ‘hand out’ memory to each process. A buggy kernel may end up handing over illegal memory addresses to processes, which would ultimately compromise the integrity of the system.

The target was simple. The kernel has complete control over the memory. If you want to access memory, ask the kernel. If you ask politely enough, you’ll get a chunk. See, simple!

Thus, I made a simple physical memory manager (page frame allocator) followed by a virtual memory manager, which provided essential functions for memory allocation, deallocation, and remapping.

This phase wasn’t without its struggles! When I say dealing with memory, it kind of means tracking every addressable byte in memory in software. A little slip up somewhere meant that I had to spend hours debugging the kernel to find the culprit. I had to control the linking process even more intimately. In case you don’t believe the struggles, there was a bug that ended up treating code as data and ended up modifying the semantics of the kernel itself! You wouldn’t want your program behaving unexpectedly because it modifies itself when it runs, would you?

Higher Half Kernel Map

The presence of memory management features allows the developer to exploit the entire virtual memory address space of the processor. It also allows the kernel to map and remap virtual addresses to any physical address.

Following this, I initialized the kernel properly and used the virtual memory manager to map it to the virtual address 0xC000000. This is pretty standard, and many modern operating systems do this “higher half” kernel map. A higher half kernel is used for consistent interrupt handling when multitasking in user — mode. With this mapping implemented, multi-tasking is now a possibility, and something which we plan on adding for the next version of KSOS!

Interrupt Handling

This corresponds to the implementation of appropriate software to correctly handle the interrupt features of the hardware system. A stable implementation of interrupt handling would lead to efficient use of hardware resources like timers, the keyboard, etc. in our operating system.

I had to spend quite a lot of time reading the actual Intel manuals to understand how interrupts worked on the x86 architecture. Implementation of good interrupt routines requires a sound understanding of assembly programming too — doesn’t everything? 😉

Moreover, reading the ~5000-page Intel Manual is HARD and requires experience! Fret not, if you properly spend time reading and researching about such low-level development, you will eventually understand the relevant texts you need from the manual.

Keyboard Driver

As soon as the kernel was handling interrupts, I wrote a simple keyboard driver to get user input and interaction. It is here that my OS development journey saw the inclusion of some more travelers! A few members of the IIT Roorkee ACM Student Chapter joined in to add more features to the driver, such as handling backspace, arrow keys, shift, caps lock, etc.

This keyboard driver was quite simple! All that was needed to be done was ‘translation’ of information about key presses and releases sent by the keyboard into whatever form we wanted. For example, if the keyboard sends a code that says ‘letter A pressed’, we interpret that the user wants to type the letter ‘A’ and output the ASCII code for ‘A’.

Kernel-Level Shell

The first version of KSOS (v1.0.0) uses all of the features explained above to implement a simple kernel-level shell with basic functionality. Something simple, awesome, and fun!!

Conclusion

This journey has been quite fulfilling, with the release of the first version of KSOS finally bearing fruit for the hard hours invested in the past several months (yay!). I want to thank the fantastic people at #osdev on freenode IRC as well as the OSDev community for their patience and support. Almost none of this would have been possible without their help! Also, I am forever indebted to the good people of StackOverflow. Last but not least, I would like to thank members of the IITR ACM Student Chapter for their help, guidance, and support.

You can run and test KSOS via this repository. There’s still a lot of work left to do, especially in areas of process management (multitasking!) and allowing users to write their very own code modules. Do feel free to contribute! We’ll also be releasing blogs highlighting the various core aspects of KSOS in the coming weeks, so stay tuned for more!

Until next time!
Suraaj