Introduction

In the last post, we saw that nwsoc was dying in pain because it relied on some wired IOCTLs. If we were, hypothetically, developing an exploit to pwn the office printer (to print some sick memes, of course) now, we would have a problem. We need nwsoc because it has some network interfaces exposed, but we can’t fuzz it because it relies on some data from the kernel (e.g., ink levels). By hooking IOCTLs, we can provide the same data that the kernel module would.

I want to focus on adding IOCTLs to QEMU. To do this, I have divided the topic into four steps:

1. Using nandsim to create a virtual mtd device. We can issue IOCTLs created by the module;
2. Build a simple program that uses this IOCTL. This program works on our guest system, but fails in qemu;
3. Patch QEMU. The demo program now runs without any issue;
4. Hook the IOCTL. Feed arbitrary data to the program.

nandsim seems to be the perfect candidate for this demo, as it’s a good compromise between setup simplicity and real-world applicability. Many embedded updaters take direct control of the flash memory, and this subsystem is not too complicated to be a burden for the scope of the post. I wanted to use nwsoc, but there is too much reverse engineering effort, and I wanted to keep the post concise.

1 - Nandsim

Here is the documentation about nand devices in Linux. We are interested in the very last part, the nand simulator.

modprobe nandsim first_id_byte=0x20 second_id_byte=0x78

This command will create a virtual 128MiB device. Remember that this memory is allocated, and will remain so until we unload the module.

2 - Simple Test Program

Ok, now we have a new device, it should be /dev/mtd0. Just make sure you’re not touching something important, especially if you’re using a Chromebook.

Now, we need some code. Below is an example of a simple program that issues an IOCTL. Most programs you’re likely to encounter are more complicated, but this does the job for our QEMU project.

This code is pretty simple:

1. Allocate space for mtd_info_t struct on the stack
2. Open a file descriptor to the virtual MTD device
3. Issue an IOCTL asking for info about the memory
4. Print some stuff back

The rest is just error handling code.

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <sys/ioctl.h>
#include <mtd/mtd-user.h>
 
int main(int argc, char *argv[]){

    if(getuid() != 0){
        printf("Not run as root!\n");
        printf("This program will likely fail!\n\n");
    }

    printf("Trying to open %s\n", argv[1]);

    mtd_info_t mtd_info;

    int fd = open(argv[1], O_RDWR);

    if (fd == -1 & errno == EACCES){
        printf("Wrong permission for device\n");
        printf("Check filename or run as root\n");
        exit(-1);
    }

    int rc = ioctl(fd, MEMGETINFO, &mtd_info);
    if (rc == -1){
        printf("Something went wrong with the IOCTL.\n");
        exit(-1);
    }
    printf("MTD total size : %u bytes\n", mtd_info.size);
    printf("MTD erase size : %u bytes\n", mtd_info.erasesize);
 
    return 0;

}

We can compile it with gcc dump_info.c -o dump_info and run it. If you have done everything right, you should get something like this:

➜ sudo ./dump_info "/dev/mtd0"
Trying to open /dev/mtd0
MTD total size : 134217728 bytes
MTD erase size : 16384 bytes

No, you’re not alone. The first time I saw that command, I questioned my math. As it turns out, 134217728 bytes is exactly 128 MebiByte, better known as 128MiB.

If we run this in QUEMU, everything breaks. Look:

➜ sudo qemu-x86_64 dump_info /dev/mtd0
Trying to open /dev/mtd0
Unsupported ioctl: cmd=0xffffffff80204d01
Something went wrong with the IOCTL.

𝓈𝒶𝒹, but we’re here to fix it.

3 - Patching QEMU

To go ahead, you need a working QEMU build environment. Doing this on arch is a nightmare, but a simple docker ubuntu container can help. You can find this docker container in the git repo for this post. I usually mount the qemu source folder as a volume and use docker to build. YMMV.

Adding support for an IOCTL that is also supported by your host system is failry easy:

1. We need to register a new IOCTL in ioctl.h IOCTL(MEMGETINFO, IOC_R, MK_PTR(MK_STRUCT(STRUCT_mtd_info_user)))
2. We need to register a new target in syscall_defs.h #define TARGET_MEMGETINFO TARGET_IOR('M', 1, struct mtd_info_user)
3. We need to define a new data type, in this case a struct, in syscall_types.h

STRUCT(mtd_info_user,
       TYPE_CHAR,
       TYPE_INT,
       TYPE_INT,
       TYPE_INT,
       TYPE_INT,
       TYPE_INT,
       TYPE_LONG
)

4. Because we used mtd_info_user in the 2nd step, we have to add #include <mtd/mtd-user.h> in syscall.c, otherwise the build will fail

Now we should be able to run our program inside QEMU without any crash:

➜  sudo qemu-x86_64 dump_info /dev/mtd0
Trying to open /dev/mtd0
MTD total size : 134217728 bytes
MTD erase size : 16384 bytes

4 - Hooking Stuff

So far, we managed to add support for a new IOCTL. Inside do_ioctl, there’s all the logic that QEMU uses to forward IOCTLs to the host system. We insert a trap to prevent QEMU from sending a specific IOCTL to the host. This approach enables us to feed arbitrary data to the application that is running in the emulator.

case TYPE_PTR:
        arg_type++;
        target_size = thunk_type_size(arg_type, 0);
        switch(ie->access) {
        case IOC_R:
            if(cmd == 0xffffffff80204d01){
                gemu_log("MEMGETINFO Hook Trigger\n");
                argptr = lock_user(VERIFY_WRITE, arg, target_size, 0);
                mtd_info_t temp_info =  {0,  0, 31337, 16, 16, 16};
                thunk_convert(argptr, &temp_info, arg_type, THUNK_TARGET);
                unlock_user(argptr, arg, target_size);
                break;
            }
            ret = get_errno(safe_ioctl(fd, ie->host_cmd, buf_temp));
            if (!is_error(ret)) {
                argptr = lock_user(VERIFY_WRITE, arg, target_size, 0);
                if (!argptr)
                    return -TARGET_EFAULT;
                
                thunk_convert(argptr, buf_temp, arg_type, THUNK_TARGET);
                unlock_user(argptr, arg, target_size);
            }
            break;

We can use this method for different tasks. For instance, because QEMU is no longer redirecting IOCTLs to the host, the guest application no longer needs a special file to run. Since we control the data that the guest application receives, potentially, we can mock a full kernel module inside QEMU.

➜  sudo qemu-x86_64 dump_info not_mtd_device 
Trying to open not_mtd_device
MEMGETINFO Hook Trigger
MTD total size : 31337 bytes
MTD erase size : 16 bytes

As you can see, the program works without any issues. We successfully hooked the IOCTL, and we can feed the program arbitrary data.

Conclusion

I really want to thank Andrea and Paolo for helping me write this post.

I also want to point out that what I described is a naive implementation. For instance, to emulate a kernel module, you need to add several other components, mostly to do state keeping. Also, keep in mind that in in the case of nwsoc I had to reverse engineer most of the functionality so it’s not a fast process.

I hope you found this useful. The code is on Github!