>
Open Source

eBPF in practice: bpftrace, bpfilter, and libbpf-bootstrap

Extending a Linux box used to leave you with two uncomfortable options. Patch the kernel and maintain a fork forever, or ship a kernel module and accept that one bad pointer takes the whole machine down with it. There is a third option now, and the interesting part is not the pitch. The interesting part is that the kernel refuses to accept your code until it has checked a short list of things about it first.

That checking step is why eBPF (an in-kernel execution facility that grew out of the Berkeley Packet Filter) is worth a sysadmin’s attention even if you never write a line of it by hand. The tools built on top of it are already good enough to install on a work server, and they answer questions that used to require a debugger, a maintenance window, and a nervous colleague watching over your shoulder.

What the kernel agrees to run

The tradeoff at the center of eBPF is capability for safety. A kernel module can do more, because a module is effectively a piece of the kernel. Programs loaded through eBPF live under restrictions the kernel enforces, and per the write-up those restrictions are the point: short of a bug in the kernel itself, a buggy program of yours is not supposed to be able to bring the system down.

Two things make that guarantee real. The first is the verifier (the kernel component that inspects a submitted program before it is allowed to run), which rejects infinite loops and illegal operations. The second is that anything which clears the verifier gets compiled by a JIT (just-in-time compiler) into native machine code for the host architecture, so the safety story does not come with an interpreter tax at every hook.

Execution is event driven rather than continuous. Where your program runs depends entirely on the hook it attaches to. Two hooks come up constantly in practice:

  • XDP (eXpress Data Path). Runs at the network interface driver level, before packets reach the kernel network stack.
  • Syscall tracepoints. Fire on calls like open(), unlink(), and fork(), which is how tracing tools see process behavior.
  • Maps. Shared data structures such as arrays, queues, and hash maps, used to pass data between the in-kernel program and its user-space companion.
  • The loader. A user-space program that hands the compiled object to the kernel and can stay resident to collect results.

Reach for the source article’s own framing here and it calls eBPF “JavaScript for the kernel,” a marketing line rather than a technical claim. Treat it as shorthand for dynamic, sandboxed, and loadable at runtime, not as a statement about the language you write in.

Two tools to try before writing any code

Most people who benefit from this never compile anything. The packaged tools do enough.

On the filtering side, bpfilter translates network filtering rules into eBPF programs and loads them for you. It began life as a drop-in replacement for iptables, and it ships a command line utility called bfcli that installs and manipulates filtering chains. The example in the source article loads a chain on the XDP hook that drops ICMP, which blocks ping, attaches it to an interface by ifindex, then flushes the chain to undo the change. The attach step wants an interface index, which ip a will give you.

Filtering at the XDP level is worth understanding as a performance decision, not just a syntax choice. Packets are dropped at the driver, before the network stack processes them, which is exactly why the article names XDP as a common pick for DDoS mitigation. If you only want that layer and nothing else, there is a narrower project called xdp-filter that focuses on XDP alone.

On the observability side, bpftrace takes small programs in an AWK-like language and turns them into eBPF programs internally. The language was inspired by the D language from DTrace, the older UNIX tracing system, so anyone who has written D will recognize the shape. A one-liner against the sys_enter_rmdir tracepoint prints a line every time any process removes a directory, and you stop it with Ctrl+C like any foreground command.

That single example is a decent way to judge whether this is useful to you. If your usual answer to “what deleted that directory” is a shrug and a grep through logs, one bpftrace line replaces the shrug.

What writing one by hand actually involves

Going below the packaged tools means two programs instead of one. The in-kernel piece compiles to an ELF binary (the standard executable format on Linux systems) that carries eBPF bytecode instead of machine code, and Clang is the usual compiler. The user-space piece submits it through the bpf() system call, with helper libraries available in C, Go, and Python.

Doing that from scratch is the part nobody enjoys. The setup code and build plumbing are, in the article’s words, tedious to write and hard to get right. Three frameworks exist to absorb that work: BCC (BPF Compiler Collection), libbpf-bootstrap, and eunomia-bpf. The article picks libbpf-bootstrap on the grounds that it stays portable and has support from some of the kernel eBPF maintainers, which is a reasonable default if you expect the code to outlive the experiment.

libbpf-bootstrap is a collection of working sample programs with the setup code and a Makefile already written, licensed BSD 3-Clause at the time the article was published. The minimal example needs git, make, gcc, clang, and libelf-dev present, then a recursive clone, a make minimal, and a run under sudo. The loader prints a dot every second and waits; the in-kernel program’s output shows up in a second terminal reading trace_pipe under the tracing debugfs path.

Look at what the sample program is attached to and the design becomes obvious. Its section annotation names the sys_enter_write tracepoint, and write() is what sits underneath higher level output functions like printf() and fwrite(). That one hook could observe output from every process on the box, which is why the shipped example includes a filter restricting it to the loader’s own process ID. The article’s suggested exercise is to repoint the section at sys_enter_rmdir and drop the process filter, which reproduces the earlier bpftrace one-liner the hard way.

Doing it the hard way is still worth an afternoon. The bpftrace version is shorter, but the framework version is where the ceiling is higher.

Tooling for when it misbehaves

Three utilities come up in the article as the day-to-day kit:

  • bpftop. Real-time view of running eBPF programs, in the spirit of top and htop.
  • bpftool. Inspection and manipulation of loaded programs and their maps.
  • llvm-objdump. Disassembles eBPF bytecode when you need to read the generated instructions.
  • trace_pipe. Not a tool exactly, but the file where bpf_printk output lands, and the first place to look when a program loads and appears to do nothing.

Editorial note, not from the source: install bpftool before you need it. Diagnosing a program that loaded but is not firing is much easier when you can list what is attached where.

Trade-offs

The safety guarantee costs you freedom. Kernel modules can do more, and the article is direct about that; if your problem needs capabilities the verifier will not allow, eBPF is the wrong tool and no amount of framework choice fixes it.

Verifier friction is the other real cost. The article notes that stricter verification produces false alarms, and that this makes development frustrating enough that improving the verifier is an active area of interest. Potential gaps in verification are a standing concern from the other direction too, which is a polite way of saying the safety story rests on the verifier being correct.

There is also a scope question worth being honest about. Ready-made tools cover filtering and tracing well, so if that is your need, your work stops at installing bpfilter or bpftrace. Hand-written programs mean a build toolchain, a framework dependency, and two programs to maintain instead of one. Tooling like Cilium sits at yet another level: the article describes it as a high-performance Container Networking Interface for Kubernetes that replaces the kube-proxy and iptables path, which is a platform decision rather than a weekend install.

Hardware and platform reach cut both ways as well. Per ebpf.io by way of the article, large operators including Google, Meta, Cloudflare, and Netflix use eBPF for something, some SmartNICs support it in hardware, and Microsoft has been working on Windows support. That breadth is a good sign for longevity and a bad reason to assume any given feature exists on the kernel in front of you.

What I would tell past me

Start at the top of the stack and only go down when something forces you to. Install bpftrace, run the directory-removal one-liner, and see whether the answers it gives are answers you actually wanted. If they are, the next step is bpfilter for filtering work, and libbpf-bootstrap after that when a packaged tool cannot express what you need.

Skip the reading list until you have run something. The article points at ebpf.io and the eBPF foundation for background, and both are more useful once you have watched a hook fire on your own machine. One tracepoint, one terminal reading trace_pipe, and the abstraction stops being abstract.

Leave a comment