TL;DR

This post shows an example eBPF Tracepoint program and shares tips on writing the eBPF Tracepoint programs with aya. 🦀🐝

Introduction

Dataflow

We will run the eBPF program where tracingpoint’s data flows from kernel space to user space like below.

Kernel space

The eBPF Program trace_sched_process_exec is invoked by tracepoint sched_process_exec. Then, trace_sched_process_exec parses the tracepoint’s data and sends it to the eBPF Map events.

User space

User space Program observer observes the eBPF Map. If observer detects new data in the map, it observer reads the data from the map.

Benefits of using aya

Before diving into codes, let’s take a look at the benefits of using aya for eBPF projects.

One language: Rust

The biggest benefit of using aya is that both kernel-space and user-space programs are written in Rust. This means the kernel-space and user-space programs can share codes!

In this example case, trace_sched_process_exec and sched_process_exec are written in Rust and compiled into binaries by rustc. And, a shared library is placed common folder.

Auto-generating Rust codes

The document says ‘Aya is an eBPF library built with a focus on operability and developer experience.‘ So, it has several tools that help developers. Aya generates code for user-space codes for loading the eBPF into the kernel and attaching it to events. Also, aya-tool generates rust struct codes.

Run eBPF Tracepoint tracing program

Let’s run my eBPF Tracepoint tracing program on your machine!

Prerequisites

Linux
Rust nightly
bpf-linker
bindgen-cli
bpftool

(Optional) Set up environment on MacOS

If you’re using MacOS, you can quickly set it up with lima and my template.

Install lima

brew install lima

Download the template

wget https://raw.githubusercontent.com/yukinakanaka/aya-lab/main/lima-vm/aya-lab.yaml

Edit cpu and memory configuration in aya-lab.yaml. Default values are:

cpus: 4
memory: "8GiB"

Create a VM

limactl start lima-vm/aya-lab.yaml

Clone

You can get all the codes from my repo.

git clone https://github.com/yukinakanaka/aya-lab.git
cd aya-lab/tracepoint-sched-process-exec

Generate structs codes

cargo xtask codegen

Build and Run

Build and execute the program!

cargo xtask build && sudo ./target/debug/observer

You can see logs of both the kernel-space program and the user-space program. These logs show information that is extracted from the tracepoint!

Example:

2024-07-06T13:33:30.024915Z  INFO observer: 54304 502 /usr/bin/sed 12
2024-07-06T13:33:30.024994Z  INFO trace_sched_process_exec: 54304 502 /usr/bin/sed 12    
2024-07-06T13:33:30.026210Z  INFO observer: 54305 502 /usr/bin/cat 12
2024-07-06T13:33:30.026252Z  INFO trace_sched_process_exec: 54305 502 /usr/bin/cat 12

Note: In the above log, the observer’s logs are older than the trace_sched_process_exec’s logs. This doesn’t mean the observer detects events earlier than the trace_sched_process_exec. This reversal of the log’s order occurs because aya-log also uses eBPF Map to send kernel space’s logs to user space as well as events. This transmitting may be slower than the events transmitting.

Tips on writing eBPF codes

I learned a lot by writing the above eBPF Tracepoint program. Let me share some tips on writing eBPF codes.

Tip1: Deserializing with generated struct codes

We must deserialize a context (ctx) to get kernel call parameters from the tracepoint events. Manually deserializing is a bit cumbersome and might be a cause of bugs. So, we should do it automatically as long as possible.

You can see generated struct codes of sched_process_exec’s parameters in ebpf/src/bindings.rs.

#[repr(C)]
#[derive(Debug)]
pub struct trace_event_raw_sched_process_exec {
    pub ent: trace_entry,
    pub __data_loc_filename: u32_,
    pub pid: pid_t,
    pub old_pid: pid_t,
    pub __data: __IncompleteArrayField<::aya_ebpf::cty::c_char>,
}

This file was generated by aya-tool.

aya-tool generate trace_event_raw_sched_process_exec

Also, aya-tool can be used in codes like xtask/src/codegen.rs. By doing that, we can run it via cargo.

cargo xtask codegen

Once you have sched_process_exec’s struct, you can deserialize it easily!

let event: trace_event_raw_sched_process_exec =
        ctx.read_at::<trace_event_raw_sched_process_exec>(0)?;

Tip2. Decode __data_loc

I struggled to get the filename from __data_loc_filename. In Rust, its type is u32_. I could not understand that meaning. Therefore, I conducted the following search.

Checked the C/C++ expression:

cat /sys/kernel/tracing/events/sched/sched_process_exec/format

name: sched_process_exec
ID: 255
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;

        field:__data_loc char[] filename;       offset:8;       size:4; signed:0;
        field:pid_t pid;        offset:12;      size:4; signed:1;
        field:pid_t old_pid;    offset:16;      size:4; signed:1;

print fmt: "filename=%s pid=%d old_pid=%d", __get_str(filename), REC->pid, REC->old_pid

The filed type of filename is __data_loc char[] and its size is 4, which means 32bit.

Read ‘[iovisor-dev] Accessing __string fields in tracepoints‘. It says

Lower 16bit is the offset against the beginning of the event entry.
Higher 16bit is the length of the array.

Using the above information, I managed to decode __data_loc by the following.

let mut buf = [0u8; 32];

let offset = (event.__data_loc_filename & 0xFFFF) as usize;
let len = (event.__data_loc_filename >> 16 & 0xFFFF) as usize - 1; // -1 is for null-termination.
let bytes = bpf_probe_read_kernel_str_bytes(
        ctx.as_ptr().add(offset) as *const u8,
        &mut buf,
)?;
let filename = core::str::from_utf8_unchecked(bytes),

Tip3. PerCpuArray eBPF Map as a Heap

When I built eBPF programs, I faced the following error Building the eBPF program failed with:

cargo xtask build-ebpf

error: linking with `bpf-linker` failed: exit status: 2
  |
  = note: LC_ALL="C" PATH="/home/
...
  = note: error: LLVM issued diagnostic with error severity

It looks like an error related LLVM, but the error message didn’t tell the cause. So, I had no idea what was wrong with my code. I could not find a solution in Google and GitHub. But, I finally found the cause by searching Aya Community’s discord!

We cannot use big variables in stack space because eBPF programs are limited to 512 bytes of stack space! I used a big variable like let buf = [0: 512]. I can solve that problem using PerCpuArray eBPF Map as a heap. Here are snippets.

#[map(name = "data_heap")]
pub static mut DATA_HEAP: PerCpuArray<ProcessExecEventBuf> = PerCpuArray::with_max_entries(1, 0);

#[repr(C)]
pub struct ProcessExecEventBuf {
    pub p: ProcessExecEvent,
}

let buf:  &mut ProcessExecEventBuf = unsafe {
        let ptr = DATA_HEAP.get_ptr_mut(0).ok_or(0)?;
        &mut *ptr
};
buf.p.pid = event.pid;

Note: The above error message is from v0.9.2 bpf-linker. It may be improved in the future.

Tip4: Inspect eBPF programs/maps with bpftool

Sometimes I had errors while loading eBPF programs and using eBPF Map. Then, I needed to check programs and maps in the kernel. In that case, bpftool is a great tool.

Check what eBPF programs are loaded into the kernel

sudo bpftool prog list

You can see trace_sched_pro!

791: tracepoint  name trace_sched_pro  tag 8d60832b8d7402c8  gpl run_time_ns 50976378 run_cnt 986
        loaded_at 2024-07-04T21:17:24+0900  uid 0
        xlated 3744B  jited 3480B  memlock 4096B  map_ids 690,693,689,691,692
        pids observer(474946)

Check what eBPF maps in the kernel

sudo bpftool map list

You can see events, data_heap.

690: percpu_array  name data_heap  flags 0x0
        key 4B  value 528B  max_entries 1  memlock 8192B
        pids observer(474946)
...
692: perf_event_array  name events  flags 0x0
        key 4B  value 4B  max_entries 8  memlock 4096B
        pids observer(474946)

Helpful references

The eBPF and aya are relatively new technologies, and they are actively developed now. So, it may be difficult to find solutions when you face problems. Bellows are really helpful references, so check them!

Aya Community’s discord

Aya community is so active that you can find helpful information on the discord! You can join it from here.

How to write an eBPF/XDP load-balancer in Rust

Kong’s blog explains how to write eBPF/XDP load-balancer using in Rust from scratch. You can learn how to set up a Rust project and generate Rust codes using aya-tools, etc. If you haven’t used Aya yet, I highly recommend that you go through this post!

suidsnoop

Uses Aya and eBPF LSM programs to implement audit logging and policy enforcement for suid binaries. I learned from it how to access a PerCpuArray eBPF Map from user space.

aya-examples

There are so many examples of aya. I learned from it how to access a PerCpuArray eBPF Map from kernel space.

Frequently asked questions about using tracepoint with ebpf/libbpf programs

This post doesn’t use aya. But, we can learn how to get kernel call parameters from the tracepoint events.

Wrap up

I showed my eBPF Tracepoint program and shared tips on writing the eBPF programs with aya. Also, I introduced some helpful references. I hope this post may help someone to write eBPF program with Aya! 🦀🐝

Writing eBPF Tracepoint Program with Rust Aya: Tips and Example

Introduction

Dataflow

Kernel space

User space

Benefits of using aya

One language: Rust

Auto-generating Rust codes

Run eBPF Tracepoint tracing program

Prerequisites

Clone

Generate structs codes

Build and Run

Tips on writing eBPF codes

Tip1: Deserializing with generated struct codes

Tip2. Decode __data_loc

Tip3. PerCpuArray eBPF Map as a Heap

Tip4: Inspect eBPF programs/maps with bpftool

Helpful references

Aya Community’s discord

How to write an eBPF/XDP load-balancer in Rust

suidsnoop

aya-examples

Frequently asked questions about using tracepoint with ebpf/libbpf programs

Wrap up

Published by Yuki Nakamura

Introduction

Dataflow

Kernel space

User space

Benefits of using aya

One language: Rust

Auto-generating Rust codes

Run eBPF Tracepoint tracing program

Prerequisites

Clone

Generate structs codes

Build and Run

Tips on writing eBPF codes

Tip1: Deserializing with generated struct codes

Tip2. Decode __data_loc

Tip3. PerCpuArray eBPF Map as a Heap

Tip4: Inspect eBPF programs/maps with bpftool

Helpful references

Aya Community’s discord

How to write an eBPF/XDP load-balancer in Rust

suidsnoop

aya-examples

Frequently asked questions about using tracepoint with ebpf/libbpf programs

Wrap up

Share this:

Published by Yuki Nakamura