Introduction

In this post I’ll try to explain how to add unsupported system calls to QEMU user.

The first part will give a bit of high level context, as to answer the question why are you doing this? The second part will be about the system call that was missing and that will be implemented. The third part will be about toolchains and development environments, or more specifically how to setup an efficient workspace to experiment with QEMU. Than will follow some closing remarks and personal reflections.

A bit of context

You’re reversing some kind of embedded ARM device update. You are doing fine with static analysis, but at some point you stumble across some very messy code. It does very wired things with some strange segments. Analyzing this by hand is usually painful and very time consuming, as you have to figure both the compression/encryption algorithms and the location of the data inside the update file.

A faster approach for solving this problem is running the update and let it perform it’s job. This can be done in two ways:

1. Running the binary on instrumented hardware. Sometimes, if you’re lucky, it’s possible to dump the updated flash content. Unfortunately, this is a rare case, as most devices encrypt the flash. We’re not getting into this scenario, as it’s out of scope. Just know this is a very big can of worms.
2. Running the binary inside emulated hardware. Most of the times this is the best bet, especially if the hardware is running known uC or inside a Linux system. In the first case, common with ARM uC, you’ll end up writing a custom loader which will help executing the program in QEMU user. In the second case, you can just use QEMU user.

Let’s suppose that we decided against option one, as the hardware it’s not available. After some brief analysis we figured that the update is supposed to run inside a Linux based machine.

The next step is doing some reverse engineering to figure out things like command line parameters or environmental variables. Usually at this point you’ll end up with a prompt like this:

What problem are we facing? How do we get around this problem? What is an ioctl? How do we add them? How do we test our modifications?

Please keep in mind that the reminder of the post will be about flash memory ioctl. I decided to go with this specific topic for several reasons, first and foremost because I believe in explanation by example. Clear and compelling examples are often better than lengthy theoretical discussion. Flash memory ioctls are a compelling example because Linux can emulate flash memories, which lets you try the following even without custom hardware. Another reason is that the whole memory subsystem is very small, which makes this a very interesting exercise if you want to learn how to navigate Linux codebase. This is an essential skill if you want to add unsupported syscalls to qemu.

Honestly, there’s also the fact that I actually had to implement this, so I had most of the stuff ready.

Syscalls, IOCTL, MTD and friends

Recall the snippet above, what did the error message mean?

Unsupported ioctl: cmd=0xffffffff80204d01

To better understand this message we need to digress a bit into syscalls and ioctl.

syscalls

Syscalls issue commands to the kernel. Syscalls are needed because not every resource can be accessed directly. A simple example is reading the content of a file. Every programmers knows that to read a file you first call open, which will return a file descriptor, followed by a read, which hopefully will return your data. The kernel takes the burden of understanding which file you want and where it’s data is. It will perform these tasks with the help of a filesystem, which in most cases will be ext4. Syscalls are cool, with them you don’t need to worry about filesystems, disks or permission. If you can do it, the kernel will help you do it. If you can’t your syscall will fail. As simple as that.

Unfortunately, it’s not so simple. The concept of a file as a container of information is really handy, because it lends very well both to the idea of user created data and to the idea of device data.

Want to store some cat pictures? Store them in a file. Want to store the current configuration your flash memory? Well, a file also works well for that. If you heard the saying that in Linux everything is a file this is what it means. A file it’s just a concept for unique data container.

This unfortunately complicates things quite a bit, because we need to distinguish between files that are used to store data and files that are used to store state of hardware devices. While the kernel tries to minimize the difference between normal files and device files, for some operations, mainly direct interaction with hardware, the distinction is so big that a new systemcall is needed.

ioctl

This is the new magic syscall that asks the kernel to control device specific functions. According to Wikipedia a good example of ioctl is CD-ROM disk ejection, as a ioctl is issued to the CD-ROM kernel module which then triggers the right sequence of commands to open the tray.

Want to issue a specific command to a hardware device in your computer? Most of the times you will have to mess with ioctl. This is great, because with just one system call the kernel can address a wide variety of device commands, without complicating the general purpose interface. In comparison, Windows has a very different approach, where every device driver effectively creates new system calls. very messy. so uncool.

This is very relevant to our hypothetical goal of reversing an embedded updater. We need to know about system calls and ioctl, because flash memory management has a interface based on ioctl which has not been implemented in QEMU, thus the error above. In the next paragraph I’ll discuss briefly about the MTD subsystem.

MTD

Most embedded systems will need a low power low cost data storage solution. Even small mechanical HD are way to expensive for most embedded applications. This used to be one of the limiting factors of embedded technologies, but thanks to the development of solid state based storage solutions this is no longer a problem. There are mainly two types of solid storage solutions. We will focus only on raw flash.

Raw flash is handled in Linux via the Memory Technology Device, MTD for short, that decouples the flash chip driver from filesystems and other utilities, making both software independent form one another.

Ioctl and MTD met at /dev/mtdX which is a character device managed by mtdchar (this is a concise way to refer to those special files meant for hardware handling that we discussed above) . It provides ioctls to manipulate and erase blocks in the underling flash memory.

Please refer to this blog post by Michael Opdenacker for more details regarding flash memories and Linux.

Friends

So, as anticipated earlier, we now can fully understand QEMU’s error message.

Unsupported ioctl: cmd=0xffffffff80204d01

As ioctl are meant to deal with specific hardware commands, QEMU doesn’t implement all of them. But thankfully it’s not to hard to add new ones in. In simple terms, QEMU maps ioctl from the guest to the host semi transparently via a set of macros. If the host system has a different endianess than the guest, you will need to create a helper function to translate the commands between the two systems, thus making the translation semi transparent. Another interesting case is when the host system doesn’t have the required ioctl. This can happen with custom peripheral. In this case you will have to mock the required ioctl in qemu, as in cannot be passed to the host system.

Conclusion

This post is getting long, so instead of going forward I will tackle this problem in some feature posts. Stay tuned (: