github/scientist
Scientist
Scientist is a Ruby code parity testing library and production experimentation framework. It allows for the safe deployment of candidate code paths alongside a control implementation to verify that new logic produces the same outputs and exceptions as the original.
The library identifies behavioral divergences between legacy and refactored code by running both versions in a live environment. It functions as a refactoring regression detector, measuring performance parity and detecting mismatches using real-world data without affecting the end user.
The system covers broad capabilities for monitoring and observability, including the publication of execution metrics and timing data to external systems. It provides tools for validating error consistency, sanitizing observation values, and applying custom comparison logic to filter known discrepancies.
Features
- Dual-Path Execution - Runs a control and one or more candidate implementations of the same logic to compare their outputs for parity.
- Production Experimentation Frameworks - A tool for safely deploying candidate code alongside a control path to detect mismatches in real-world environments.
- Logic Divergence Detection - Validates new logic against real-world production data to detect behavioral mismatches without affecting users.
- Performance Regression Analyzers - Tracks execution time and CPU usage across multiple implementations to identify performance regressions.
- Deployment Regression Detectors - Identifies behavioral regressions in production by comparing execution behavior between software versions.
- Refactoring Regression Detectors - Identifies behavioral divergences between legacy and refactored code by executing both in production.
- Code Path Interception - Wraps critical code paths to capture return values and exceptions from multiple versions for side-by-side validation.
- Custom Comparison Logic - Implements user-defined comparison functions to determine if different return values or exceptions are functionally equivalent.
- Experimentation Frameworks - Provides a framework for safely deploying candidate code alongside control paths to identify mismatches in production.
- Parallel Implementation Migrations - Facilitates safe code migration by comparing new logic against original implementations in a live environment.
- Surgical Code Refactoring - Enables precise, verified replacement of production code paths by ensuring identical results between versions.
- Behavioral Divergence Detectors - Detects behavioral divergences between legacy and refactored code by running both side-by-side during execution.
- Error Consistency Validation - Compares exceptions raised by different code paths to ensure new implementations fail in the same way as the original.
- Cross-Implementation Parity Verifiers - Validates that different software implementations of the same logic produce identical outputs and exceptions.
- Implementation Parity Testing - Executes a new code path alongside a control path to verify that both return the same values and perform similarly.
- Parity Mismatch Detection - Raises an exception when two code paths diverge to identify correctness issues during automated test runs.
- Performance Measurement - Tracks execution and CPU time for multiple code paths to identify performance regressions during a refactor.
- Ruby Parity Testing Libraries - Provides a specialized Ruby library for executing and comparing two versions of a code path.
- Ruby Refactoring Utilities - Offers Ruby-specific tools for running multiple versions of a code path to compare performance and results.
- Conditional Execution Gating - Provides mechanisms to control whether experimental code paths run based on custom flags to manage production rollout risk.
- Experiment Execution Controls - Provides flags to determine whether candidate code paths should execute during production rollouts.
- Experiment Result Publishing - Sends timing data and execution metadata to custom destinations for external analysis of results.
- Analysis Result Filtering - Applies custom logic to ignore known discrepancies or simplify return values for easier analysis of differences.
- Observability Data Filters - Suppresses reports of specific mismatches based on provided conditions to reduce noise in experimental data.
- Metric and Performance Monitors - Sends execution timing and mismatch data to external monitoring systems for performance and correctness analysis.
- Candidate Version Comparison - Runs a single control implementation against several alternative versions to compare different approaches simultaneously.
- Mismatch Filtering - Suppresses known discrepancies using custom predicates to reduce noise during experimental validation.
- Regression Testing Suites - Provides a side-by-side execution workflow to ensure new implementations do not introduce functional regressions.
- Code Analysis and Metrics - Library for safely refactoring critical code paths.
