bpftrace/bpftrace
Bpftrace
bpftrace is a high-level eBPF tracing tool and kernel instrumentation framework for Linux. It provides a tracing language to instrument kernel and user-space events without recompiling the system, functioning as a dynamic system profiler and event aggregator.
The project enables dynamic system tracing and Linux kernel observability by capturing tracepoints and dynamic probes in real time. It allows for kernel data inspection and runtime process debugging by accessing internal data structures and filtering specific process events.
Its capability surface covers system performance analysis, including execution stack sampling and event latency measurement. It also provides utilities for event data summarization through histograms and frequency maps, as well as a probe discovery utility to locate available instrumentation points.
Features
- Dynamic System Tracing - Enables adding instrumentation to a live system using eBPF to observe kernel and user-space function calls.
- Data Structure Inspectors - Allows accessing and summarizing internal kernel data structures to diagnose complex system behaviors.
- Runtime Type Resolution - Provides dynamic type resolution using BTF and kernel headers to cast raw memory addresses into readable C structures.
- eBPF Tooling - Provides a high-level tracing language that leverages eBPF to instrument kernel and user-space events.
- Event Aggregators - Summarizes system event data into histograms and frequency maps for deep performance analysis.
- Kernel Observability Programs - Provides advanced observability of the Linux kernel by capturing internal events and inspecting system state in real time.
- Kernel Structure Tracing - Provides the capability to access and dereference internal kernel structures by casting raw arguments or using type information.
- Kernel-Userspace Shared Maps - Utilizes shared hash maps and arrays between the kernel and userspace to aggregate and store event counters.
- Kernel Tracing Frameworks - Provides a framework for capturing tracepoints and dynamic probes to analyze kernel data structures and function calls.
- Kernel Event Tracers - Captures event data using stable tracepoints and dynamic probes at function entry and exit points.
- Dynamic Kernel Probes - Allows attaching custom eBPF bytecode to kprobes and tracepoints for live kernel function instrumentation.
- Performance Profiling Tools - Functions as a dynamic system profiler that samples execution stacks and measures event latency.
- System Performance Analyzers - Measures event latency and execution stacks to identify performance bottlenecks and resource contention in running systems.
- Execution Stack Samplers - Captures execution stack traces at a set frequency to identify system performance bottlenecks.
- Event Frequency Counters - Aggregates event occurrences into maps, frequency counts, and histograms for performance trend visualization.
- Runtime Debugging - Filters and captures specific process events to troubleshoot application crashes or unexpected behavior on a live host.
- Predicate-Based Execution Filters - Executes actions only when specific boolean expressions or predicates are true to isolate targeted process IDs.
- Source Translators - Implements a backend that translates high-level tracing scripts into C code before eBPF compilation.
- Just-in-Time Compilers - Translates high-level scripts into eBPF bytecode at runtime using an LLVM-based just-in-time compiler.
- Program Lifecycle Managers - Manages the lifecycle of kernel programs by loading bytecode and reading data from shared memory maps.
- Event Latency Measurements - Implements high-resolution timestamping to measure the time elapsed between two specific events.
- Probe Discovery Utilities - Provides a utility to discover available instrumentation points using search terms and wildcards.
- Command Line Tools - Listed in the “Command Line Tools” section of the The Book Of Secret Knowledge awesome list.
