jepsen-io/jepsen
Jepsen
Jepsen is a distributed systems testing framework designed to verify the correctness of distributed software. It functions as a fault injection engine and a distributed consistency analyzer that validates whether a system's behavior matches its promised consistency guarantees by analyzing operation histories against formal models.
The project distinguishes itself through its ability to simulate complex failure scenarios, including network partitions, node crashes, and clock drifts. It employs packet-level traffic shaping to drop or delay communication and manipulates system time across clusters to identify bugs related to clock skew and synchronization errors.
The framework provides a full orchestration surface for deploying nodes across virtual machines, containers, or physical hardware. Its capability set covers the entire verification workflow, from managing remote command execution and node roles to performing linearizable history analysis and rendering operation timelines via web-based interfaces.
The system persists complete execution histories and performance metrics to disk, allowing for offline auditing, anomaly hunting, and the calculation of latency and throughput under stress.
Features
- Distributed System Verification - Introduces faults and analyzes operation histories to ensure a distributed system behaves exactly as specified.
- Linearizability Analyzers - Analyzes the total order of operations to ensure the system adheres to strong consistency guarantees.
- Fault Injection Testing - Coordinates controlled system failures like network partitions and node crashes to test resilience.
- Network Faults - Introduces network faults by dropping, delaying, or shaping packets to test communication resilience.
- History Correctness Verification - Validates recorded operation histories against a formal model to detect consistency violations and correctness errors.
- Clock Drift Simulations - Simulates clock skew and time jumps across clusters to identify synchronization bugs in distributed systems.
- Consistency Model Verifiers - Checks if a system's behavior matches promised consistency guarantees by analyzing operation histories.
- Distributed Consistency Analyzers - Validates recorded operation histories to detect linearizability violations and data loss in distributed clusters.
- Time Manipulation - Stubs or offsets the system clock to simulate time jumps and drifts across a distributed cluster.
- Operation History Analysis - Analyzes recorded logs of system operations to detect anomalies and generate correctness reports for distributed systems.
- Distributed Systems Testing - Verifies that distributed software remains correct and consistent when subjected to network partitions and node failures.
- Model-Based Consistency Verifiers - Validates recorded operation sequences against a formal specification to detect consistency violations.
- Interactive Debugging and Testing - Loads serialized test data and operation histories into an interactive environment for debugging and anomaly hunting.
- Storage IO Fault Injection - Induces filesystem faults by flipping bits or truncating files to simulate data loss and hardware failure.
- Test Cluster Deployers - Deploys and manages distributed nodes across virtual machines, containers, or physical hardware for testing.
- Traffic Shaping - Controls network communication by dropping or delaying packets to simulate unstable connections.
- Event Logging - Persists complete execution histories of operations and faults to disk for offline auditing and anomaly hunting.
- Resilience Benchmarks - Measures latency and throughput while a system is under stress or experiencing partial failures.
- Clock Drift Visualizers - Tracks and plots clock drift over time to identify timing anomalies across distributed nodes.
- Operation Timeline Visualization - Generates a visual timeline of the execution history to audit the exact sequence of system events.
- Clock Skew Analyzers - Measures and simulates clock skew and drift across multiple nodes to find synchronization errors.
- Performance Analysis - Calculates latencies and throughput rates to evaluate system behavior under stress via rendered graphs.
- Test Orchestration - Coordinates server setup, failure injection, and operation execution to interpret the final results of a test run.
- Infrastructure and Systems - Framework for verifying distributed systems.
- Distributed Systems Frameworks - Framework for verifying distributed systems through fault injection.
