brendangregg/perf-tools
Perf Tools
This project is a specialized toolset for profiling kernel latency, analyzing tracepoint frequency, and monitoring system-wide performance data. It functions as a kernel performance profiler, tracepoint analyzer, and a collection of utilities for the Linux ftrace and perf_events subsystems.
The toolkit provides high-level abstractions via shell-scripted wrappers to manage complex kernel tracing interfaces. It distinguishes itself through the use of bucket-based event histograms to visualize the distribution of kernel events and the ability to identify functions exceeding specific latency thresholds using a function graph profiler.
The capability surface covers several diagnostic domains, including disk I/O latency and request size analysis, network TCP retransmission tracing, and page cache hit and miss monitoring. It also includes tools for dynamic instrumentation via kprobes and uprobes, as well as process tracking for system calls, signal delivery, and execution.
Features
- Kernel Event Tracers - Implements non-intrusive instrumentation frameworks for capturing system-wide kernel events and function execution data via ftrace.
- Network and Execution Tracing - Captures and analyzes network traffic and system calls to monitor block I/O and process execution in real time.
- Kernel Tracepoint Histograms - Creates summaries of tracepoint variables inside the kernel using multiple counting points and configurable bucket ranges.
- Kernel Event Analyzers - Aggregates tracepoint data into histograms and summaries to analyze the frequency of specific kernel events.
- Ftrace Utilities - Provides a set of tools for tracing kernel events, function calls, and system behavior via ftrace.
- Event Aggregators - Provides utilities to summarize kernel event data into histograms and frequency maps for distribution analysis.
- Performance Counter Sampling - Collects system performance data by sampling hardware and software counters using the Linux perf_events subsystem.
- Kernel Tracing Frameworks - Records executions of kernel functions matching wildcards and filters them by process or thread ID to analyze behavior.
- Kernel Execution Profilers - Visualizes kernel-level timing and resource usage to identify system bottlenecks using perf and ftrace.
- System Call Tracing - Provides real-time monitoring of kernel system calls, including arguments and return values.
- Event Distribution Histograms - Generates distributions of tracepoint variables using configurable bucket sizes to analyze the frequency of specific kernel events.
- Function Latency Tracing - Identifies kernel functions exceeding a specific latency threshold using a function graph profiler to find bottlenecks.
- Dynamic Kernel Probes - Ships tools that attach to live kernel functions by patching binary code to capture real-time events.
- Linux Performance Analysis - Offers a collection of scripts for analyzing performance metrics and profiling kernel functions on Linux.
- User-Space Function Tracing - Instruments specific memory addresses in user-space binaries to monitor function calls without modifying executables.
- Page Cache Analysis - Tracks cache hit and miss statistics by profiling kernel functions for page accesses, writes, and additions.
- Page Cache Profiling - Profiles page cache hits and misses to evaluate memory efficiency and filesystem performance.
- I/O Latency Tracers - Captures block-level events to monitor process IDs, request types, device offsets, and latencies.
- TCP Packet Drop Analysis - Identifies and diagnoses the causes of TCP packet drops and retransmissions to analyze network instability.
- Retransmission Tracing - Monitors packet retransmits and loss probes to provide details on process IDs and socket states.
- System Call Aggregators - Aggregates the number of system calls executed across the system or by specific processes for a set duration.
- Signal Delivery Tracing - Monitors system calls to identify which process sends signals to which ID and records the operation result.
- Memory Access Profilers - Profiles memory access patterns and page cache statistics to detect performance bottlenecks.
- Static Tracepoint Filtering - Enables specific kernel tracepoints to print live events and stack traces filtered by process or thread ID.
- Function Call Counters - Tracks how often specific kernel functions execute over a set time interval to identify high-traffic events.
- Process Lifecycle Tracing - Monitors the execution of new processes and their arguments to capture short-lived tasks.
- Disk I/O Performance Analysis - Analyzes block device latency and request sizes to diagnose storage bottlenecks and queuing delays.
- System Diagnostics - Performance analysis tools based on Linux perf and ftrace.
- Command Line Tools - Listed in the “Command Line Tools” section of the The Book Of Secret Knowledge awesome list.
- System Performance Tools - Collection of performance analysis tools using ftrace and perf.
- System Profiling Tools - Performance analysis based on Linux perf and ftrace.
