What eBPF Means for Observability vs. Security
The Extended Berkeley Packet Filter (eBPF) has seen a steady uptick in adoption as the technology has evolved for more than 10 years. Since its creation, it has served as a par excellence way to distribute code directly from the kernel and through a sandbox throughout the network.
It enables the gathering of information and interaction with code and applications on the network, essentially covering the full scope of any code or application controlled by or connected to the kernel.
Given eBPF’s extensive coverage across environments, including Kubernetes, eBPF has been a boon for observability, security monitoring and networking. Between observability and security, there is some overlap, but the use cases with eBPF are largely distinct. For reasons we’ll explore below, eBPF has not yet reached its full potential for either case.
Many of eBPF’s functionalities arguably fall under the categories of security and observability. eBPF can filter out noise while monitoring a large number of file operations, Bill Mulligan, a Cilium and eBPF community pollinator for Isovalent at Cisco, told The New Stack.
“eBPF for observability is often the easiest entry point, providing immediate visibility with minimal risk,” Mulligan said. “But its impact goes far beyond that. It’s reshaping how we secure and operate modern infrastructure, even letting us do things like customizing scheduling for better gaming performance.”
eBPF is already being used to enforce security policies in production, from filtering syscalls (seccomp BPF) or enabling runtime security enforcement through projects like Tetragon, Mulligan said. For example, he noted, Google uses it for most of its kernel security: “The key is using eBPF appropriately.”
Indeed, he said, eBPF is significantly more efficient than traditional security mechanisms like kernel modules or user-space polling.
“However, like any tool, it needs to be applied thoughtfully. High-frequency events, frequent user-space transitions, and complex map lookups can introduce overhead if not optimized properly,” Mulligan said. “The key is balancing performance and protection by leveraging [Linux security model] hooks, selective event tracing and efficient data structures to minimize impact while still enforcing strong security policies.”
eBPF and Observability
How eBPF benefits observability is pinned to how the observability providers make use of it to help organizations achieve their observability goals. Their methods will differ based on the specific needs of the organizations using them, ensuring the balance between data collection and manageability.
However, the right tools and platforms must take increasingly nuanced approaches, as some organizations may or may not require visibility into the kernel, while others will seek solutions that might not require data monitoring and scraping on the kernel level.
The risk is that eBPF data collection can become a firehose of logs and metrics that can overwhelm users seeking to analyze events or when debugging. Resources such as OpenTelemetry (OTel) can be used to help contextualize the data before observability solutions are fully implemented. OTel, a standard set of practices for collecting observability data, plays a key role in organizing and reducing the overwhelming influx of data generated by eBPF.
OTel for eBPF metrics and observability in general has benefited from Elastic’s donation of the Elastic profiling agent to the OTel project. Monitoring and observability provider Elastic is committed to further integrating the Elasticsearch project with OTel in a major way, recognizing it as a vital cornerstone for enhancing users’ search experiences.
By leveraging Elasticsearch for data searches and observability across diverse databases and data types, users can benefit from its versatility and robustness. OTel plays a pivotal role in this integration, serving as a central component in enabling seamless data monitoring and analysis across various environments, from the kernel and throughout the network.
eBPF and Security
eBPF offers great support for security in many but not all use cases. Given the overhead in certain situations, it may be better to continue using eBPF to gather information to enhance overall observability, but the actual implementation for security would create too much overload.
The difference between a security and an observability tool that reports on events is “that a security tool needs to be able to distinguish between events that are expected under normal circumstances and events that suggest malicious activity might be taking place,” Liz Rice, head of open source at Isovalent, writes in her book “Learning eBPF,” published by O’Reilly.
A case example Rice described in her book is an application that writes data to a local file as part of its normal processing. While the app is expected to write to /home/<username>/< filename, this activity “isn’t something you’re interested in from a security perspective,” Rice wrote.
However, she added, the user would want to be notified if the app writes to one of the many sensitive file locations in Linux: “It is unlikely that it needs to modify the password information stored in /etc/passwd.”
In some use cases, eBPF might appear to serve as a security function, but in reality, it does not. In some cases, it could even create security holes when eBPF is used to gather information for security fixes.
As Rice cited in her book, a hacker window was discussed in a DEFCON 29 talk, “Phantom Attack: Evading System Call Monitoring.” Additionally, the manual page for seccomp_unotify explicitly notes that, “It should thus be absolutely clear that the seccomp user-space notification mechanism cannot be used to implement a security policy!”
In Gartner’s report, “Hype Cycle for Monitoring and Observability, 2024,” analyst Simon Richard writes: “eBPF improves observability, security and performance for applications. However, most enterprises will not use eBPF directly. Technology vendors do use eBPF as an underpinning technology in their products and services to improve the performance and safety of programs that run on Linux.
“eBPF allows extremely technically savvy organizations to safely and quickly make changes to Linux. It is an improvement over alternative approaches, such as using Linux kernel modules or upstreaming to the Linux distribution.”
Gartner’s recommendation that “extremely technically savvy organizations” can benefit from eBPF for observability points to how its adoption and application can pose challenges, especially if an in-house eBPF expert team is required to instrument eBPF directly.
“I don’t think vendors should simply adopt eBPF blindly,” Lin Sun, head of open source at Solo.io, and a Cloud Native Computing Foundation technical oversight committee member and ambassador, told The New Stack. “Vendors should choose what is the best fit to solve the problems they have based on the skills and usability of a technology.”
As Gartner notes, eBPF is difficult and requires technical skills and know-how to implement it directly, especially without the abstraction layers for its use that observability and security providers might offer.
Its deployment in hyperscalers offered by major cloud providers such as Amazon Web Services (AWS), Microsoft Azure and Google Cloud, remains largely transparent to the user. While eBPF is by now commonly in use in any observability provider’s offering, garnering actionable insights depends on where it is applied for observability.
In the case of service mesh, it is better to use the appropriate alternative. Relying instead on iptables rules might make more sense for observability and security for service mesh on Kubernetes, Sun said.
“There is not much improvement we observed when replacing iptables with eBPF,” she said. “And doing so could introduce additional requirements.”
