Linux KCOV for Dummies (like me)

KCOV is a wonderful tool with a very simple premise: instrument kernel code with calls to functions that spit out their return addresses so that a developer can understand which code paths have been reached by a given test case. The biggest use (that I have dealt with) for KCOV is as a form of guidance for coverage guided kernel fuzzers such as Syzkaller. By running test cases on target VMs, the fuzzer can extract coverage from KCOV and decide which test cases are "interesting." That is, test cases that provide new coverage, or hit coverage points in different orderings. I'm also working in a project in which we want to maximize code coverage and use our own instrumentations to understand memory accesses under typical use. This lead to custom KCOV collection in multiple execution contexts and a rabbithole of chasing what little documentation there was. I figure I'll put all of what I learned so far together in one place so it's easier for others to find.

Anyways, no guarrantees that everything here will be exactly correct, but it should be good enough to create a basic understanding and get you started.

Background

To get started, I'll introduce a few ideas and definitions that I will use in this writeup.

First, KCOV is specifically Linux kernel coverage. Its goal is to tell you what portions of kernel code your test cases covered. In searching around, I did find a tool that collects coverage from user-space programs called KCOV, but this is not the same thing.

Second, basic blocks are segments of code that have a single entry point and a singe exit point, and no branches. They are handy for coverage because if we find that part of a basic block was executed (say from a KCOV instrumentation), we know the entire basic block was executed. KCOV will use this fact and put its instrumentations at the beginning of each basic block in the kernel.

Third, PCs, RIP, and return addresses. A PC is a program counter. It generically refers to the address of an instruction in memory. Sometimes when talking about KCOV, we say we are collecting PCs. We are talking about the addresses of KCOV instrumentation points. RIP is the x86 CPU register (EIP if you're using 32 bits) that holds the address of the next instruction to execute (IP is Instruction Pointer). This is particularly important on x86 devices as each instruction varies in length. You may find this information useful when looking into PCs. Return addresses are specific instruction pointers that hold the address of the instruction to execute on return from a function. They are stored on the stack.

Fourth, Syzkaller is a wonderful coverage-guided fuzzing tool and the primary reason I'm dealing with KCOV. It's goal is to generate and mutate random test cases and maximize the coverage it is able to collect from them. It is responsible for finding several thousands of bugs in the Linux kernel since 2017 and is also a great example of how to implement coverage collection using KCOV. Check out the project here.

Fifth, DWARF is a debug format that is compiled into the kernel alongside the actual executable code. Basically, it tells us where the compiler put things such as whether a function was inlined or how code was rearranged. Note, it isn't perfect. Sometimes DWARF says one thing, but the compiler did a different thing after that step (optimization steps are generally good, but tricky). If you use something like nm or addr2line, its probably using DWARF under the hood. I will probably make a writeup focussing solely on DWARF at some point here.

Finally, for the project that motivated this, I'm using Linux 6.8.1. To make things a little easier, my example code is based on 6.8.0. This matters to you because Linux KCOV has changed slightly since this version. I'll leave it to you to make sure your work works one whatever version of Linux you choose.

Compile-Time Instrumentation

KCOV is a used as a compile-time instrumentation, meaning the compiler handles putting the sanitizer calls in code (Technically, KCOV is a sanitizer, but it is not used to manifest crashes in the same way as KASAN or UBSAN). Both GCC and more recently Clang (note, the exact versions required to compile Linux with KCOV change as newer versions are released. As of 6.8.0, gcc-12+ and clang-13+ are recommended) support the flag:

-fsanitize-coverage=trace-pc

This tells the compiler that you would like it to insert calls to the function:

void __sanitizer_cov_trace_pc();

By default, the sanitizer calls are inserted at the beginning of each basic block. This means that each basic block, and thus each line of code, can be mapped to a single coverage call, and eventually a unique PC. However, this part only handles the function insertion into the binary (Or IR code if you want to get fancy). The responsibility of defining the function __sanitizer_cov_trace_pc() still falls to the developer. In the Linux source code, it is defined in kernel/kcov.c as follows:

/*
 * Entry point from instrumented code.
 * This is called once per basic-block/edge.
 */
void notrace __sanitizer_cov_trace_pc(void)
{
    struct task_struct *t;
    unsigned long *area;
    unsigned long ip = canonicalize_ip(_RET_IP_);
    unsigned long pos;

    t = current;
    if (!check_kcov_mode(KCOV_MODE_TRACE_PC, t))
        return;

    area = t->kcov_area;
    /* The first 64-bit word is the number of subsequent PCs. */
    pos = READ_ONCE(area[0]) + 1;
    if (likely(pos < t->kcov_size)) {
        ...
        WRITE_ONCE(area[0], pos);
        barrier();
        area[pos] = ip;
    }
}
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);

The function is a robust, yet lightweight way of saying: "If KCOV_MODE_TRACE_PC is enabled for this thread, write the return address of this function to some shared memory. Otherwise return faster than a boomerang on Red Bull."

And that is how the instrumentation end of KCOV works. Linux has defined the sanitizer function, and a developer can (somehow) tell the compiler through a flag to insert the function calls into the kernel during compilation. Next we'll cover how to configure Linux to use KCOV.

Kernel Config

The Linux kernel config (Kconfig) is a (very large) set of environment variables which describe how the kernel Makefile should compile the kernel. The config is defined in the .config file at the root of the kernel directory. For starters, you can run make defconfig or find your host config at /boot/config-$(uname -r) (at least on Ubuntu). For concision, here's what the useful Kconfig looks like related to KCOV:

CONFIG_ARCH_HAS_KCOV=y
CONFIG_KCOV=y
CONFIG_KCOV_ENABLE_COMPARISONS=n
CONFIG_KCOV_INSTRUMENT_ALL=y
CONFIG_KCOV_IRQ_AREA_SIZE=0x40000
CONFIG_DEBUG_FS=y
# CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
CONFIG_DEBUG_INFO_DWARF4=y
# CONFIG_RANDOMIZE_BASE is not set

These configs are defined in lib/Kconfig.debug.

CONFIG_ARCH_HAS_KCOV is selected by the architecture (x86/amd64 in my case) when it is able to use CONFIG_KCOV. If it says no, you're probably out of luck.

CONFIG_KCOV exposes the internal function __sanitizer_cov_trace_pc() (as well as the rest of kernel/kcov.c) and allows for KCOV instrumentation. Note, this config does not perform said instrumentation.

CONFIG_KCOV_ENABLE_COMPARISONS is an optional method of colecting coverage that places additional instrumentations around comparisions and branch statements. In addition to the return address, each KCOV call also gives the operands od the comparison that was performed in the binary. This method also uses different functions void __sanitizer_cov_trace_cmpX(); and compiler flag -fsanitize-coverage=trace-cmp. Note, we won't use this in this post, but some fuzzers do use it.

CONFIG_KCOV_INSTRUMENT_ALL enables KCOV instrumentation for the whole kernel. If you want to do generic coverage-guided fuzzing, this is the option you want. If you want to manually instrument the kernel with KCOV, you probably want to turn this off.

CONFIG_KCOV_IRQ_AREA_SIZE: quoting the Kconfig documentation, "KCOV uses preallocated per-cpu areas to collect coverage from soft interrupts. This specifies the size of those areas in the number of unsigned long words." I think that explains it perfectly.

CONFIG_DEBUG_FS enables the filesystem at /sys/kernel/debug, which we are going to use to interact with KCOV.

CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT: DWARF debug information is basically the compiler taking notes on everything it did during compilation. The most important thing is that it tracks what functions were inlined and what binary instructions came from which line of code. This allows us to figure out where in source code the PCs we collect from KCOV came from. You'll note I've turned this one off. I have found that only one DWARF toolchain can be enabled at a time, and I specifically want DWARF4 turned on for use with Syzkaller and some other tools. I suggest finding the toolchain that works for you. This post isn't concerned with reading DWARF directly, but perhaps I will make another post on how to do that for specific applications. For Linux versions older than 5.12, use CONFIG_DEBUG_INFO.

CONFIG_DEBUG_INFO_DWARF4: As I mentioned above, I specifically want this DWARF toolchain enabled for compatibility with Syzkaller and some other tools.

CONFIG_RANDOMIZE_BASE is basically KASLR (Kernel Address Space Layout Randomization). I won't comment here whether it is actually useful in defending against attacks, but it does make our lives much more difficult when trying to gather coverage or do any other debugging or sanitizing. Turn it off. There are patches that are supposed to make KCOV work even with this on, but it is much easier to have complete knowledge of the layout and for it to be consistent.

Finish building your kernel with make olddefconfig and make -j`nproc`.

Collecting Coverage

Now for the reason I made this post. There is exactly one file's worth of documentation provided by Linux on KCOV. You can find it here. I do suggest reading it.

Local Coverage

For starters, coverage collection is a per-thread process. As we saw above, the function __sanitizer_cov_trace_pc() checks if KCOV has been enabled for this thread, and if not returns. So, to collect coverage, we have to know what threads we want to collect coverage from, and then enabel KCOV on those threads. For local coverage, the workflow is to enable coverage, make some syscall, then read the coverage output and disable coverage again. When we make the syscall, the thread maintains its thread ID (PID/TID) and coverage remains enabled as execution passes into kernel space, so KCOV knows to output coverage (Have a look at the task_struct).

Let's walk through the user-space code.

#include <stdio.h>
#include <stddef.h>
#include <stdint.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <fcntl.h>
#include <linux/types.h>

#define KCOV_INIT_TRACE   _IOR('c', 1, unsigned long)
#define KCOV_ENABLE       _IO('c', 100)
#define KCOV_DISABLE      _IO('c', 101)
#define COVER_SIZE        (64<<10)

#define KCOV_TRACE_PC  0

The includes are for C libraries that you'll use and the definitions are exactly the same as include/uapi/linux/kcov.h. The COVER_SIZE variable can be increased based on your requirements.

int fd = open("/sys/kernel/debug/kcov", O_RDWR);
if (fd < 0)
    perror("open"), exit(1);

We open this file under debugfs in order to create a channel to export the coverage data and communicate with KCOV using ioctl's. If this file does not exist on your system, try mounting debugfs: mount -t debugfs none /sys/kernel/debug.

if (ioctl(fd, KCOV_INIT_TRACE, COVER_SIZE))
    perror("ioctl$KCOV_INIT_TRACE"), exit(1);

This ioctl sets up KCOV in trace mode and allocates the coverage buffer in kernel memory. We'll mmap it as shared memory in a moment. Check the source code of the ioctl here.

unsigned long *covermem = (unsigned long*)mmap(NULL, COVER_SIZE * sizeof(unsigned long), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if ((void*)(*covermem) == MAP_FAILED)
    perror("mmap"), exit(1);

This sets up the shared memory in user-space. In order to read the output of KCOV, we just read this memory.

if (ioctl(fd, KCOV_ENABLE, KCOV_TRACE_PC))
    perror("ioctl$KCOV_ENABLE"), exit(1);
__atomic_store_n(&cover[0], 0, __ATOMIC_RELAXED);

Now we turn on KCOV for this thread by setting the mode to KCOV_TRACE_PC. Once again the source code relating to the ioctl is here. The result of this is that if we every call into kernel-space from this thread and call into the function __sanitizer_cov_trace_pc(), then KCOV will write the return address of that function into the shared memory we set up. The call to __atomic_store_n() is how we reset the shared memory in order to collect new coverage. Technically speaking, you don't have to call it here, but if we instrumented the entire kernel with KCOV, we just got a little bit of uninteresting coverage from the tail end of the ioctl.

Now, make your syscalls! Remember, KCOV is only enabled on this thread, so your syscalls should directly follow the KCOV_ENABLE ioctl. If you want to get fancy, you can setup KCOV in one thread (up through mmaping the shared memory), then fork a child, enable KCOV in the child, and make your syscalls from there. This would enable you to loop the collection of coverage in the aprent thread using the shared memory you set up. Yes, this would end up being concurrency race central, but you can always run test cases more than once if your think you missed any coverage.

unsigned long n = __atomic_load_n(&(*covermem)[0], __ATOMIC_RELAXED);
for (i = 0; i < n; i++)
    printf("0x%lx\n", covermem[i + 1]);

Above is a simple way to read the coverage. Load the entries from the shared memory and print them out in hex. I'll put a more complex way below for if you are collecting coverage from a child thread.

if ((n = __atomic_load_n(&(*covermem)[0], __ATOMIC_RELAXED)) == 0)
    continue;

unsigned long *work_cover = (unsigned long *)malloc(COVER_SIZE * sizeof(unsigned long));
memcpy(work_cover, *covermem, n);
__atomic_store_n(&(*covermem)[0], 0, __ATOMIC_RELAXED);

for (i = 0; i < n; i++)
    if (work_cover[i + 1] != 0)
        printf("0x%lx\n", work_cover[i + 1]);

free(work_cover);
work_cover = NULL;

The idea here is to check if anything has been written to the shared memory and how much, then copy that data to a working buffer and reset the coverage buffer for the next time around. Then print out all non-zero entries and reset. Again, this is super race-prone, but the only thing you'll miss is a bit of coverage. Run the test cases a few more times and you will likely see all the coverage you expect.

Real quick about finishing up coverage collection. If you are collecting from the same thread (that is, no forks or execs), you can close everything down as follows:

if (ioctl(fd, KCOV_DISABLE, 0))
    perror("ioctl$KCOV_DISABLE"), exit(1);

if (munmap(*covermem, COVER_SIZE * sizeof(unsigned long)))
    perror("munmap"), exit(1);

if (close(fd))
    perror("close"), exit(1);

However, it may be easier to let the program die. KCOV cleans up nicely after itself once the thread terminates, which is nice if you've forked and exec'd and have no clean way to clean up.

Remote Coverage

Sometime you want to collect coverage from more than just the local thread. The Linux kernel is full of background threads in multiple different contexts (worker threads, softirq, etc.). As you may have noticed, the KCOV instrumentation is still put in those code blocks, but we need a way to tell KCOV to export the coverage to us. To start, we're going to define a few more constants, a struct, and a function.

struct kcov_remote_arg {
    __u32           trace_mode;
    __u32           area_size;
    __u32           num_handles;
    __aligned_u64   common_handle;
    __aligned_u64   handles[0];
};

#define KCOV_INIT_TRACE             _IOR('c', 1, unsigned long)
#define KCOV_REMOTE_ENABLE          _IOW('c', 102, struct kcov_remote_arg)

#define KCOV_SUBSYSTEM_COMMON       (0x00ull << 56)
#define KCOV_SUBSYSTEM_USB          (0x01ull << 56)
#define KCOV_SUBSYSTEM_MASK         (0xffull << 56)
#define KCOV_INSTANCE_MASK          (0xffffffffull)

#define KCOV_COMMON_ID              0x42
#define KCOV_USB_BUS_NUM            1

static inline __u64 kcov_remote_handle(__u64 subsys, __u64 inst)
{
    if (subsys & ~KCOV_SUBSYSTEM_MASK || inst & ~KCOV_INSTANCE_MASK)
        return 0;
    return subsys | inst;
}

There's a lot going on here, so let's just go over each item when we use it. For now just take a moment to look at each of these and understand what they do code-wise. Again, many of these definitions can be found here.

Start your coverage collection code the same as you did for local coverage collection up through the shared memory allocation. Do note that if you want to collect both local and remote coverage, you need to set up shared memory for each of them. The setup is copied below:

int fd = open("/sys/kernel/debug/kcov", O_RDWR);
if (fd < 0)
    perror("open"), exit(1);

if (ioctl(fd, KCOV_INIT_TRACE, COVER_SIZE))
    perror("ioctl$KCOV_INIT_TRACE"), exit(1);

unsigned long *covermem = (unsigned long*)mmap(NULL, COVER_SIZE * sizeof(unsigned long), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if ((void*)(*covermem) == MAP_FAILED)
    perror("mmap"), exit(1);

Now let's take a detour into remote handles. KCOV uses identifiers called handles to figure out whether coverage in remote threads should be turned on. In kernel code, you can find calls to the function void kcov_remote_start(u64 handle), which is generally called from its wrappers found in include/linux/kcov.h.

In this function, KCOV checks to see if the handle passed to kcov_remote_start() has been registered, and if so, turns on KCOV for that thread. Then, at some later point, kcov_remote_stop() is called to turn off KCOV again. Note, kcov_remote_start/stop() are called manually by kernel developers. They are not placed at compile time and you can find them in the source code. For instance, kcov_remote_start_common() is called at the start of every vhost worker thread.

So, what the heck is a handle? Every handle is a 8 byte identifier made up of a subsystem and an id. The upper 1 byte is the subsystem, of which there are currently two valid values: KCOV_SUBSYSTEM_COMMON and KCOV_SUBSYSTEM_USB. The lower 4 bytes are the id, which are defined by the kernel developer or the KCOV user (who may both be the same person). Bytes 4-7 are never used and must be 0 for the handle to be valid. Check out kcov_check_handle() and the two masks we defined above: KCOV_SUBSYSTEM_MASK and KCOV_INSTANCE_MASK.

The id part of the handle is a little tricky I found. For common handles, we (the KCOV user) will pass an id to the current task struct, which will in turn get propagated to background threads spawned by our current thread. So we can make it whatever we want in user space and it will get propagated down. For homework, go check out the vhost_worker() code and see that the kcov_handle field is taken from the current task struct. You can see above we defined #define KCOV_COMMON_ID 0x42. For the USB subsystem, the id is the USB bus number. So if you wanted to look at bus 1, you'd use the id 1 (just like we did above for KCOV_USB_BUS_NUM).

KCOV_SUBSYSTEM_COMMON and KCOV_SUBSYSTEM_USB are the only two subsystems currently defined in the kernel. If you want to collect remote coverage from a different subsystem, say IPv6, you'll need to modify the kernel. I'll cover this in a future section.

For now, back to our user-space code.

struct kcov_remote_arg *arg;
arg = (kcov_remote_arg*)calloc(1, sizeof(*arg) + sizeof(uint64_t));
if (!arg)
    perror("calloc"), exit(1);
arg->trace_mode = KCOV_TRACE_PC;
arg->area_size = COVER_SIZE;
arg->num_handles = 1;
arg->common_handle = kcov_remote_handle(KCOV_SUBSYSTEM_COMMON, KCOV_COMMON_ID);
arg->handles[0] = kcov_remote_handle(KCOV_SUBSYSTEM_USB, KCOV_USB_BUS_NUM);
if (ioctl(fd, KCOV_REMOTE_ENABLE, arg))
    perror("ioctl$KCOV_REMOTE_ENABLE"), free(arg), exit(1);
free(arg);

Enabling coverage collection for remote coverage is a bit more complicated compared to local coverage. For one, it takes a kcov_remote_arg struct, which we have copied from kernel space (Alternatively, if you're familiar with vmlinux.h, that should do the trick). Allocate enough space for the struct as well as the array that's hanging off the end. You'll note we told the array it was of size 0, and now we just tack on some extra memory and call it good. Don't you just love memory unsafe languages? Anyways, that's what the extra + sizeof(uint64_t) is at the end of the allocation. The trace mode is the same as before: KCOV_TRACE_PC, as well as the cover size. Now we get to the remote handles. The struct has room for the common handle, as well as any number of uncommon handles (yeah, that's what their called in kernel code). Since the only two subsystems taht exist right now are common and USB, we just need the 1 extra handle. num_handles cares only about the uncommon handles, so set it to 1 for now.

We'll use the function kcov_remote_handle() to make each handle by passing the subsystem identifier and a chosen ID. So for common, that's KCOV_SUBSYSTEM_COMMON and the ID KCOV_COMMON_ID. Remember, that ID is going to get passed to the task struct, so it can be whatever we want.

Our USB handle, determined by the subsystem KCOV_SUBSYSTEM_USB, is going to use the USB bus number as the ID. In the example, we chose bus 1 (KCOV_USB_BUS_NUM), but if you wanted to observe more busses, you could add more handles.

Finally, we can enable the remote coverage by calling the KCOV_REMOTE_ENABLE ioctl and passing our remote arg struct we worked so hard to craft. Once coverage is enabled, we can free the arg struct without any worry. Then, just read and disable the remote coverage once you're done with it. If you want to see KCOV done properly, check out how the Syzkaller project does it.

Making your own remote KCOV

Sometimes (most of the time), you'll find that using the established subsystems isn't enough to collect the KCOV information you want. For instance, nearly the entire recieve side of the IPv6 stack is in softirq. Right now, there's no way to collect coverage from that code in mainline Linux. The solution? Do it yourself. In my recent project, I needed to collect remote coverage from the IPv6 module, which we were building and instrumenting separately from the rest of the kernel. This will be a brief recounting of the steps I took to get it working. My method was pretty simple, so it isn't going to be the best method for every application.

For this project I focused on two types of threads: worker threads and softirqs. Worker threads ended up being pretty easy for this project, mostly because they run in process context and I could use process_one_work() as a singular entry point. Yes, that is the entry point for all workers, but in this kernel, only ipv6.ko was instrumented with KCOV, so it didn't matter. Softirqs were a little more tricky. They don't have a PID and I had to find the specific entry point I was interested in rather than using do_softirq(). Either way, let's start with the new infrastructure.

include/linux/kcov.h:
							
static inline void kcov_remote_start_ipv6(u64 id)
{
    kcov_remote_start(kcov_remote_handle(KCOV_SUBSYSTEM_IPV6, id));
}

static inline void kcov_remote_start_ipv6_softirq(u64 id)
{
    if (in_serving_softirq()) {
        kcov_remote_start(kcov_remote_handle(KCOV_SUBSYSTEM_IPV6, id));
    }
}

include/uapi/linux/kcov.h:

#define KCOV_SUBSYSTEM_IPV6    (0x02ull << 56)

These functions and subsystem ID lay out a new IPv6 subsystem that I decided to use for this project. You'll note that USB has the same infrastructure. To finish up the subsystem, I also added it to kcov_check_handle():

kernel/kcov.c:

static inline bool kcov_check_handle(u64 handle, bool common_valid,
                bool uncommon_valid, bool zero_valid)
{
    if (handle & ~(KCOV_SUBSYSTEM_MASK | KCOV_INSTANCE_MASK))
        return false;
    switch (handle & KCOV_SUBSYSTEM_MASK) {
    case KCOV_SUBSYSTEM_COMMON:
        return (handle & KCOV_INSTANCE_MASK) ? common_valid : zero_valid;
    case KCOV_SUBSYSTEM_USB:
+   case KCOV_SUBSYSTEM_IPV6:
        return uncommon_valid;
    default:
        return false;
    }
    return false;
}

Now that the new subsystem is set up, I only needed to tell KCOV where to start collecting the remote coverage. For worker threads, I only needed to make the following changes:

kernel/workqueue.c:

+#include <linux/kcov.h>
...
static void process_one_work(struct worker *worker, struct work_struct *work)
{
    ...
    lockdep_invariant_state(true);
    trace_workqueue_execute_start(work);
+   kcov_remote_start_ipv6(0x69);
    worker->current_func(work);
+   kcov_remote_stop();
    ...
}

This code is simple enough. Whenever we are about to start a worker, enable remote KCOV collection. Again, since this is a limited project, I deciced to use a constant ID for the handle. The downside to this is that since each registered KCOV handle has to be unique, only one userspace KCOV reader will be able to look for this coverage at a time. However, this works for my use case. We're looking for incoming IPv6 packets which aren't nicely tied to a userspace process, so passing something like a PID doesn't make sense. Now for softirqs. I found the entry point I was concerned with in net/ipv6/ip6_input.c.

net/ipv6/ip6_input.c:

int ipv6_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt, struct net_device *orig_dev)
{
+   kcov_remote_start_ipv6_softirq(0x69);
    struct net *net = dev_net(skb->dev);

    skb = ip6_rcv_core(skb, dev, net);
    if (skb == NULL) {
+       kcov_remote_stop_softirq();
        return NET_RX_DROP;
    }
+   int ret = NF_HOOK(NFPROTO_IPV6, NF_INET_PRE_ROUTING,
                    net, NULL, skb, dev, NULL,
                    ip6_rcv_finish);

+    kcov_remote_stop_softirq();
+    return ret;
}

This may come as a surprise, but the function do_softirq() is not itself in softirq. That fact is important because softirqs can interrupt each other, and we don't want to recursively enable KCOV. However, the best way to check for this is to use the in_serving_softirq() function, which depends on being called from softirq context. Because of this, we need to locate the actual entry function we are interested in. For me, that's ipv6_rcv(), the function that handles any recieved IPv6 packets. Aside from adding the calls to start and stop KCOV, I refactored the return value to make sure we stop KCOV at the right time. Now, this is the part you'll need to figure out for yourself. If something goes wrong where you aren't getting the coverage you expect, put calls to printk() everywhere and watch the kernel console as you go.

When we go to read the remote coverage from our newly supported subsystem, it'll look about the same as what we did for the USB subsystem, just with our constant ID:

#define KCOV_IPV6_ID 0x69

struct kcov_remote_arg *arg;
arg = (kcov_remote_arg*)calloc(1, sizeof(*arg) + sizeof(uint64_t));
if (!arg)
    perror("calloc"), exit(1);
arg->trace_mode = KCOV_TRACE_PC;
arg->area_size = COVER_SIZE;
arg->num_handles = 1;
arg->common_handle = kcov_remote_handle(KCOV_SUBSYSTEM_COMMON, KCOV_COMMON_ID);
arg->handles[0] = kcov_remote_handle(KCOV_SUBSYSTEM_IPV6, KCOV_IPV6_ID);

You can see I simply replaced the USB subsystem with IPv6.

Mapping Coverage to Lines

Figuring out what areas of source code were covered by your test cases is pretty darn simple. Let's say you instrumented the entire kernel with KCOV and built everything into it (no inserting modules at runtime). The output of KCOV probably looks something like this:

0xffffffff83269b2c
0xffffffff8333e7be
0xffffffff83445491
0xffffffff834455c6
0xffffffff83480b13
0xffffffff836fffff
0xffffffff8374f078
0xffffffff8374f0e7
...

These values are the runtime return addresses of each of the KCOV sanitizer calls your test cases hit. We can correlate these values against the actual KCOV callsites by dumping the kernel image and looking for every call to __sanitizer_cov_trace_pc() as follows:

objdump -d vmlinux | grep "__sanitizer_cov_trace_pc"

And since we are most interested in the addresses of the callsites, we can grep those alone with:

objdump -d vmlinux | grep "__sanitizer_cov_trace_pc" | grep -o "^\([0-9a-f]\+\):" | grep -o "\([0-9a-f]\+\)"

These addresses should match to the return addresses you go at runtime (possibly minus the section offset which you can check with readelf -SW vmlinux or objdump -h vmlinux). As a note, turning off KASLR is what made this possible. You can then track the PCs back to source code using addr2line as follows:

addr2line -afi -e vmlinux < [your PCs]

addr2line will give the origin function, file, and even line number of the given PCs.

Finally, if you instrumented a single module (such as ipv6.ko) and inserted it at runtime, you'll need to take into account the offset of the modules at runtime, which can be gathered from /proc/modules, as well as the simple fact that your code got relocated and you may not be able to track it back to a specific callsite in the .ko file. I found that some of the callsites I was looking for were put in the .retpoline_sites section and others were relocated to some point before the supposed module offset. I have not yet found a silver bullet for collecting coverage from modules. (I did find that for a given module, its offset is always the same, even across kernel builds).

Conclusion

I hope you found this helpful in centralizing a lot of the information about KCOV. I am prone to errors, so make sure you check your work at every step of the way. I also recommend reading the materials I linked in this writeup as well as the excellent example shown by Syzkaller, which has additional information on coverage collection in its documentation.