Nextest

nextest is a high-performance test execution framework and Rust test runner designed to manage parallelism, retries, and timeouts. It serves as a test recording system that captures execution metadata and outputs into archives for later failure analysis and replay.

The project distinguishes itself as a flakiness detection tool, identifying unstable tests through stress loops and automated retry policies. It also functions as a CI test orchestrator, capable of partitioning test suites across multiple workers and exporting results in standardized JUnit XML and JSON formats.

The framework provides broad capabilities for test execution, including process isolation to prevent shared state interference, a domain-specific language for complex test filtering, and resource slotting to prevent collisions during concurrent runs. It also covers observability through system call tracing, dynamic execution tracing, and integration with external debuggers.

The tool supports a wide configuration surface, utilizing JSON schemas for validation and providing platform-specific overrides to ensure consistent behavior across different operating systems and architectures.

Features

High Performance Task Execution - Executes tests using a high-performance execution model to maximize speed and reliability.

Rust Test Process Isolation - Runs Rust test suites with high parallelism and strict process isolation to improve reliability.

Test Execution Recording and Replay - Captures test execution data into archives to replay runs locally or analyze performance traces.

Test Result Persistence - Stores execution results on disk to enable replaying runs or selectively rerunning tests.

Test Execution Archives - Captures test execution metadata and outputs into archives for later replay and analysis.

Test Concurrency Management - Controls the number of concurrent threads used for the test suite and individual tests to balance resources.

CI Test Distribution - Distributes test loads across multiple workers and exports results in a standardized XML format.

Process Isolation - Executes each test in a dedicated process to prevent global state interference and memory corruption.

Test Execution Replays - Records execution data and replays previous runs to facilitate efficient failure analysis.

Retry Policies - Reruns failed tests using fixed or exponential backoff strategies to handle intermittent failures.

Flaky Task Retry Managers - Identifies unstable tests through stress loops and automated retry policies to ensure build quality.

Resource Slot Scheduling - Assigns unique integer slot numbers to concurrent tests to prevent resource collisions, such as network port conflicts.

System Call Tracing - Logs and filters low-level interactions between the test process and the operating system using a system call tracer.

Event Stream Recording - Captures all test execution events and metadata into a serialized archive for later analysis or replay.

Non-Deterministic Failure Detection - Identifies non-deterministic test failures through stress loops and automated retries.

Test Isolation - Executes each test in a separate process to prevent shared state interference and isolate crashes.

Flaky Test Analyzers - Identifies unstable tests by running them in stress loops to verify consistent behavior.

Test Execution Timeouts - Enforces time limits on tests and benchmarks to terminate processes that exceed defined windows.

Test Retry Strategies - Automatically reruns failed tests multiple times to identify and mark flaky test cases.

Parallel Test Execution - Runs individual tests in separate processes to isolate failures and reduce total execution time.

Concurrency Limiters - Restricts the number of simultaneous resource-intensive tests using weights to prevent system exhaustion.

Parallelism Tuning - Sets the number of simultaneous tests using CPU counts or fixed integers to balance resource usage.

Test-Level Debugging - Allows launching specific tests under a debugger or system call tracer to isolate and analyze failures.

Test Suite Partitioning - Splits a test suite into multiple buckets using slicing or hashing to execute tests in parallel.

Test Stress Loops - Identifies unstable tests by running them in repeated stress loops to detect intermittent failures.

Test Selection Patterns - Identifies tests or packages using equality, substrings, regular expressions, or glob patterns.

Rust Test Execution Engines - Provides a high-performance execution engine for Rust tests with process isolation.

Test Discovery Mechanisms - Provides automated processes for identifying and queuing all discoverable tests within a workspace.

Test Execution Filtering - Provides mechanisms to select specific tests for execution based on names, groups, packages, or target platforms.

Complex Filter Composition - Combines selection criteria using intersection, union, and negation to isolate specific test sets.

Test Execution Recording - Captures statuses, durations, and outputs into a cache for later analysis or failure reproduction.

Test Selection DSLs - Uses a domain specific language with boolean logic and regex to select specific test subsets.

Test Execution Management - Manages test parallelism, retries, timeouts, complex filtering, and resource slotting.

Machine-Readable Test Results - Produces test execution data in JSON format to facilitate integration with analysis scripts.

Target-Specific Build Settings - Adjusts configuration values based on the build target platform to optimize behavior across different operating systems.

Test Performance Profiling - Analyzes test execution times to identify and flag slow tests that may indicate performance regressions.

Test Dependency Analysis - Identifies sets of tests using predicates based on package names and dependency relationships.

Platform-Aware - Applies conditional settings and overrides based on the host or target operating system and architecture.

Test Execution Wrappers - Executes test binaries via wrapper scripts to support cross-compilation and specific target environments.

Fail-Fast - Stops the test suite after a specific number of failures to provide immediate feedback.

Platform-Based Execution Overrides - Overrides retries or timeouts based on the host or target platform using configuration expressions.

Platform-Specific Config Overrides - Applies specific configuration values based on the operating system to ensure optimal behavior across environments.

CI Test Orchestrators - Partitions test suites across multiple workers and exports results in standardized JUnit XML and JSON formats.

Execution Profiles - Groups configuration options into named profiles for applying different settings in local versus CI environments.

Test Run Replays - Reproduces results or analyzes failures by executing a specific set of tests from a prior run.

Serial Output Streaming - Passes test output directly to the terminal in serial order to prevent interleaved text from concurrent tests.

Undefined Behavior Detection - Runs tests through a specialized interpreter to detect illegal operations and memory errors.

Execution Replay - Reproduces previously captured test executions to inspect results using different reporting options.

Test Execution Prioritization - Assigns numeric weights to tests to determine their specific order of execution within the suite.

Environment Setup Scripts - Executes custom scripts before tests based on platform requirements or filters to prepare the environment.

Project Configuration - Sets project-wide rules for retries and timeouts in a shared configuration file for version-controlled consistency.

Test Resource Slots - Assigns unique integer identifiers to concurrent tests to prevent collisions when using shared resources like ports.

Execution Progress Tracking - Displays real-time test progress via progress bars and counters during execution.

Execution Tracing - Provides probes for dynamic tracing to monitor test attempts and measure execution timing in real time.

Test Configuration - Assigns specific behaviors, such as automatic retries or serial execution, to individual tests.

Test Execution Overrides - Customizes timeouts, retries, and priority for specific tests based on name filters or platforms.

Immediate Failure Termination - Halts the execution of new tests immediately after the first failure occurs.

Flaky Test Metadata Reporting - Tags tests that required retries to pass and includes this flakiness status in execution reports.

Flaky Test Outcome Policies - Determines if tests that eventually pass after an initial failure are recorded as successful or failed.

Test Result Reporters - Generates test results in JUnit XML format for integration with external reporting tools.

Test Coverage Collections - Collects execution data during test runs by integrating with coverage tools and reporting services.

Execution Standardizations - Provides a shared configuration file to standardize rules for retries, timeouts, and test groups across the project.

Test Grouping Utilities - Organizes tests into custom groups to manage shared resource usage and ensure mutual exclusion.

Failed Test Rerunning - Tracks and executes only the specific test cases that failed in a previous run across iterations.

Hanging Test Cancellations - Terminates unresponsive tests by sending system signals to the process to ensure reliable cleanup.

Interpreter-Based Testing - Uses specialized interpreters to execute tests for the purpose of detecting memory errors and undefined behavior.

Live Test Status Querying - Provides an interactive way to inspect process IDs, duration, and captured output of tests while they are running.

Subprocess Leak Detection - Detects and marks tests as failures if they leave orphaned child processes after completion.

Test and Benchmark Execution - Measures code execution speed by running benchmarks across multiple measurement harnesses.

Test Execution Pipelines - Manages a lifecycle of setup scripts, test execution, and teardown with scoped environment variable injection.

Test Process Leak Detection - Flags tests that fail to clean up child processes upon completion.

Test Setup Scripts - Executes scripts to initialize databases, services, or fixtures before tests begin with specific scoping.

Execution Prioritization - Allows assigning priority values to tests to ensure critical cases are executed first.

Cross-Platform Test Execution - Enables running test suites across different operating systems and architectures using platform-specific overrides.

Testing Frameworks - Parallel test runner with advanced filtering and output.

nextest-rsnextest

Features

Star history