The rapid evolution of eBPF (Extended Berkeley Packet Filter) has fundamentally changed the way developers think about system-level observability, performance monitoring, and security. The development and innovation pace of eBPF has been astonishing in the last years, largely due to two aspects:
- Constant Linux kernel development with features that enhance eBPF functionality and extend it to additional use cases.
- Thriving community and commercial applications that rely on eBPF.
Even operating systems like Windows are catching up, with eBPF for Windows progressing steadily.
In this article, we detail our journey in developing and optimizing the Cimon eBPF agent, a tool designed to process thousands of events per second. We focus on the engineering challenges we encounter, particularly in achieving high performance and stability in production CI/CD environments.
Key takeaways include:
- Lightning-Fast Event Processing: How to optimize the sensor to handle thousands of system events in real-time, providing rapid insights and improved observability across CI/CD workflows.
- Advanced Kernel-Level Enhancements: Implement strategies like early filtering within the kernel, precise selection of eBPF program types, and tailored map configurations to minimize latency and resource consumption.
- Robust Scalability & Security: Integrate performance monitoring and intelligent event management to not only scale to meet high-volume demands but also enhance security by tracking malicious activity and ensuring artifact integrity.
What is eBPF
eBPF is a revolutionary technology that allows developers to write sandboxed programs to run in the Linux kernel, without modifying kernel source code or loading modules. It is extensively used for performance profiling, network security, system observability, and more. Its appeal lies in its ability to run safely in kernel space, enabling developers to extract granular data, track events, and enforce policies.
The following architecture from ebpf.io illustrates how eBPF programs are loaded, verified, and attached to different kernel hooks, as well as how the user mode/kernel mode interaction happens:
Why eBPF Development is Hard
Developing eBPF programs is non-trivial and shares several challenges with traditional kernel development:
- Steep Learning Curve: A thorough understanding of Linux kernel internals is essential. Many developers find that debugging kernel-level code is significantly more complex than user-space programming.
- The eBPF Verifier: Every eBPF program must pass through a kernel verifier that checks for safety and resource limits. The verifier’s error messages are often cryptic and require in-depth knowledge of eBPF assembly, resulting in extended development effort. For instance, a seemingly redundant if/else branch might be all that is necessary for the verifier to accept the program, but discovering this often involves extensive trial and error.
- Kernel Compatibility: eBPF programs frequently interact with internal kernel structures that may change between kernel versions. While technologies like CO-RE (Compile Once, Run Everywhere) allow for dynamic checking of symbol availability, ensuring compatibility across multiple kernel versions remains a substantial engineering challenge.
- eBPF Feature Availability: The availability of eBPF features is dependent on the kernel version. For example, certain functionalities (like Ringbuf, introduced in kernel 5.8) are critical for efficient data handling. Replacing such features for older kernel versions is often non-trivial, leading to complex conditional logic in eBPF programs. An example of a source that aggregates eBPF features per Linux version can be found at: https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md.
Performance and Scale in eBPF
In cloud-native SaaS applications, scale often means handling massive user requests. In the context of eBPF, “scale” refers to the ability to ingest, process, and analyze vast amounts of low-level system events without compromising system performance or stability.
For example, consider a simple Dockerfile:
FROM ubuntu:22.04 RUN apt update && apt install figlet ENTRYPOINT ["figlet", "hello"]
Building the container image using this simple Dockerfile through docker build generates over 4,500 (!) events with filesystem read/write throughput approaching 1 GiB/s. Now, imagine orchestrating hundreds of such builds concurrently. This will require the sensor to be engineered to:
- Run in Kubernetes clusters with huge volumes of events.
- Provide support for as many event types as possible. The more event types and hooks, the more security context there will be.
- Security applications can’t afford to drop events.
- We must ensure minimal overhead to avoid disrupting production workloads.
Below are 7 key lessons we’ve learned on our path to building a high-performance, robust eBPF sensor.
1. Filter Data as Soon as Possible
The best way to handle large volumes of data is to not handle them at all – in other words, filter early. If you can filter unneeded events in the kernel, the user space will have less load to process.
In Cimon, we specifically focus on CI/CD security, so we ignore any events not originating from recognized CI/CD processes. We detect these through process names (e.g., Runner.Worker for GitHub Actions) or environment variables when there is no clear convention (e.g., JENKINS_URL for Jenkins). This keeps background OS processes from unnecessarily entering our output execution report.
The strategy can be applied to other use cases:
- Only track filesystem events for certain processes or paths.
- Filter network events by cgroup or container ID for container-specific monitoring.
- Implement more advanced hierarchical filtering (e.g., process trees, cgroups, etc.).
By doing this filtering in kernel space, we drastically reduce the overhead in user space.
2. Choose Program Types Wisely
eBPF programs are small, sandboxed pieces of code that run inside the Linux kernel, enabling dynamic extension of its functionalities without modifying the kernel source.
Different eBPF program types have varying performance characteristics. Selecting the appropriate type for your use case is essential for minimizing latency and overhead.
- Tracepoints: Pre-defined hooks in the kernel that offer stable interfaces for monitoring system events with minimal overhead. For example:
sys_enter_openatfor tracking all file open operations. You can observe all available tracepoints by exploring the tracing directory in the kernel:
> sudo cat /sys/kernel/debug/tracing/available_events ... syscalls:sys_exit_openat2 syscalls:sys_enter_openat2 syscalls:sys_exit_openat syscalls:sys_enter_openat ...
- Raw Tracepoints: Similar to tracepoints but bypass some kernel abstractions to provide even lower overhead and higher performance at the cost of less context. They are more suitable for performance-centric applications. The list of available raw tracepoints is the same and is available here:
/sys/kernel/debug/tracing/available_events. - Kprobes: Allow dynamic instrumentation of almost any kernel function by intercepting function calls. These hooks are more flexible but can introduce more overhead than tracepoints and raw tracepoints. They are suitable for situations where you need to hook into internal kernel functions that offer better context than the pre-defined tracepoints. For example, hooking into
vfs_read / vfs_write, which contains thestruct file *parameter. A list of all available Kprobes (and also Fentry hooks) is available here:
> sudo cat /sys/kernel/tracing/available_filter_functions ... vfs_readv vfs_read vfs_readlink ...
- Fentry: Introduced in Linux kernel 5.5, Fentry is a more efficient alternative to Kprobes that hooks directly into function entry points, eliminating the need for separate exit handling and thereby reducing overhead.
- BPF LSM (Linux Security Module): Allows you to attach eBPF programs to the LSM hook points in the kernel to enforce custom security policies via eBPF. This allows dynamic security checks and monitoring without modifying core kernel code. When supported by the kernel, we suggest using BPF LSM hooks for security applications that wish to identify and prevent security issues directly from the kernel. Additionally, LSM programs support special BPF helpers, including
bpf_ima_file_hash, which calculates hashes for important files using Linux kernel IMA (Integrity Measurement Architecture). Introduced in Linux kernel 5.7. - Uprobes: Like Kprobes, but target user-space applications. Allow dynamic instrumentation of user-space functions for tracing and debugging without modifying the application binaries. A main usage for such hooks would be to trace complex high-level logic that is impossible to get in kernel mode. For example, tracing HTTPS / TLS connections encrypted in user-mode libraries, such as OpenSSL. Using the
SSL_readandSSL_writefunctions in thelibssl.solibrary, you can track all encrypted payloads sent by OpenSSL.
There is this great source of performance test between Kprobes, tracepoints, and raw tracepoints:
tracepoint base kprobe+bpf tracepoint+bpf raw_tracepoint+bpf task_rename 1.1M 769K 947K 1.0M urandom_read 789K 697K 750K 755K
In general:
- Raw tracepoints > tracepoints > fentry > kprobes > uprobes, in terms of performance.
- LSM hooks are usually best for security use cases if your kernel supports them.
3. Choose Map Types Wisely
eBPF maps serve as the primary data exchange mechanism between the kernel and user space. The choice of map type can have a big impact on performance and memory usage. A few common patterns:
Example 1: Transferring Events to User Space
When you need high-throughput event delivery, Ringbuf (BPF_MAP_TYPE_RINGBUF) is ideal. It offers low latency and supports multiple producer CPUs with a single consumer in user space, preserving the order of written events.
One of the challenges is sizing the ring buffer. A large buffer reduces dropped events but consumes more memory. A small buffer risks frequent overflows. Balancing these is critical.
Example 2: Temporary Buffers
It is common to need temporary buffers for tasks such as traversing data before passing it on. In such cases, we can utilize BPF_MAP_TYPE_PERCPU_ARRAY with a single entry. This type of map is designed to be per CPU, meaning the buffer is not shared between CPUs. By defining it as per CPU, we reduce the overhead associated with synchronizing these structures.
Here is an example of how this map would be defined in code:
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
__type(key, int);
__type(value, char[BUFFER_SIZE]);
__uint(max_entries, 1);
} tmp_buffer SEC(".maps");
It can be accessed in the following way:
int zero = 0;
char *buf = bpf_map_lookup_elem(&tmp_buffer, &zero);
if (!buf) {
// Shouldn't happen
return 0;
}
...
Example 3: Lookup Tables for Hostnames
The best way to construct such a list would be to use a hash map (BPF_MAP_TYPE_HASH), where the hostname serves as the key.
For example:
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, char[HOSTNAME_MAX]);
__type(value, bool);
__uint(max_entries, 100);
} allowed_hosts SEC(".maps");
As a result, string traversal and comparison will be significantly simplified, and the insertion and lookup algorithms will be very efficient (besides the hashing algorithm, which is dependent on string length, insertion and lookup will be O(1)).
4. Monitor Your Programs for Performance Anomalies
No matter how optimized your eBPF programs are, you’ll still want to monitor them for anomalies like high CPU usage or dropped events.
Program Execution Metrics
One impressive eBPF project for gathering program metrics and monitoring CPU consumption is the open-source bpftop by Netflix. This project leverages the BPF_ENABLE_STATS syscall, which enables global runtime statistics that are disabled by default. When enabled, each eBPF program reports two key metrics: its total execution runtime and the number of executions.
These metrics allow you to calculate the average execution time, optimize performance, and identify bottlenecks. However, note that these statistics can impact the overall performance of your eBPF programs, so it is advisable to use them primarily in debugging environments.
If you are using the Cilium eBPF library, you can enable these stats with the following code:
// Enable stats
enableStats, err := ebpf.EnableStats(uint32(unix.BPF_STATS_RUN_TIME))
if err != nil {
return fmt.Errorf("failed enabling stats: %w", err)
}
defer enableStats.Close()
...
// Print stats
info, err := prog.Info()
if err != nil {
return fmt.Errorf("failed getting program info: %w", err)
}
runtime, _ := info.Runtime()
cnt, _ := info.RunCount()
runtimeNano := uint64(runtime.Nanoseconds())
var avg uint64
if cnt != 0 {
avg = runtimeNano / cnt
}
...
Monitoring Event Drops
Using a ring buffer means there is always a risk of overflow under high load. You can track dropped events directly in your eBPF code:
static __inline long send_event_with_stats(void *data, __u64 size, __u64 flags) {
// Sending the event to the ringbuf.
long res = bpf_ringbuf_output(&events, data, size, flags);
// Getting the stats map
struct ringbuf_stats *stats;
u32 zero = 0;
stats = bpf_map_lookup_elem(&events_stats, &zero);
if (stats != NULL) {
// Updating metrics
res == 0 ? stats->sent++ : stats->lost++;
stats->available = bpf_ringbuf_query(&events, BPF_RB_AVAIL_DATA);
bpf_map_update_elem(&events_stats, &zero, stats, BPF_ANY);
}
return res;
}
This information (sent and lost metrics) is eventually retrieved in user mode and published along with the rest of the program metrics information.
Monitor Map Capacity
Whenever there are maps that are constantly updated and are vital to the functionality of the software, their capacity should be monitored.
When working with ringbufs, we use the following call to determine the amount of unread data:
available = bpf_ringbuf_query(&events, BPF_RB_AVAIL_DATA);
This function retrieves the number of bytes in the ring buffer that have been written but not yet consumed, providing a real-time snapshot of buffer usage. Note that bpf_ringbuf_query is a kernel-only helper.
For other map types, capacity can be assessed by iterating through them in user space. However, such operations can be costly in terms of performance and should be limited to scenarios like debugging or diagnostic modes to avoid unnecessary overhead.
5. Do as Much as Possible in the Kernel
Kernel-level processing is inherently faster than delegating work to user space. Therefore, whenever possible, embed your logic directly within eBPF programs instead of transferring it to the user mode. For instance, we can:
- Maintain an updated process tree (based on executions and forks) in order to filter events
- Monitor the status of network connections by tracking connections and terminations.
- Correlate TCP connections with TLS connections so that TLS events can be enriched with IP and port information.
6. Manage High Bandwidth Events Efficiently
High event rates can quickly consume memory, potentially leading to system slowdowns or crashes. For instance, file analysis bandwidth can reach several GB/s, meaning even a minimal per-event memory footprint can cause your program to run out of memory in no time.
Our approach to mitigate this includes:
- Record only the information and fields necessary for analysis.
- Avoid additional memory allocations or persistent storage for events beyond the initial allocation and promptly free memory once event processing is complete.
- Instead of relying solely on the garbage collector to reclaim memory periodically, consider implementing a dedicated memory pool. For example, in Go, using sync.Pool can help manage memory more efficiently and reduce garbage collection overhead.
7. Consider Event Wakup Strategy
When producing events to a ring buffer, the final parameter in functions like bpf_ringbuf_output and bpf_ringbuf_submit (depending on the method) is __u64 flags. By default, this value is set to 0, allowing the kernel to determine when to wake up the user-space process. While this default setting works well in many scenarios, for very high-throughput scenarios, you might:
- Avoid Immediate Wakeup: Use the
BPF_RB_NO_WAKEUPflag to defer user-space wakeups, reducing interrupt overhead. - Implement Custom Wakeup Logic: For instance, trigger a wakeup only when the ring buffer reaches a certain fill level.
This trade-off requires careful tuning to balance latency against overhead.
Utilizing eBPF in Cycode
At Cimon, we recognized eBPF’s potential at the beginning of its rapid evolution, leveraging its powerful observability and prevention capabilities to monitor malicious process execution, network connections, and file system activity in CI/CD pipelines. We initially focused on hosted runners but soon expanded to self-hosted runners in diverse environments. Our ability to scale these capabilities was made possible by key eBPF ecosystem advancements, such as CO-RE (Compile Once, Run Everywhere) and BTF (BPF Type Format), which have significantly simplified eBPF agent deployment across multiple platforms.
In addition, Cimon utilizes the rich data provided by file system events to generate actionable insights, construct supply chain maps within CI/CD processes, and enhance artifact integrity. By tracking dependencies and the creation of new artifacts, we can directly link each artifact back to its source code and generate comprehensive provenance documents.
Furthermore, we deploy Cimon within Cycode to detect malicious packages and safeguard the open-source community. For example, our recent work highlighted how our system can identify malicious code hidden in npm packages. By leveraging eBPF’s deep packet inspection capabilities to analyze network traffic anomalies, we are able to flag suspicious packages and contribute to a more secure open-source ecosystem.
Summary
At Cimon, eBPF has redefined our approach to observability and security in CI/CD pipelines by enabling real-time monitoring of thousands of events per second, from capturing process behaviors and network traffic to tracking file system activities. This high-performance capability not only deepens our insights into supply chain and artifact integrity but also bolsters our ability to detect and mitigate malicious activities. Through these engineering innovations and advanced kernel-level programming techniques, we’ve pushed the boundaries of what’s possible, ultimately strengthening our overall security posture and protecting both our systems and the open-source community.
