evidentlyai/evidently

Features

• AI Observability Tracing - Captures real-time inputs and outputs from AI applications to analyze execution paths and debug system behavior.
• LLM Execution Tracing - Captures model-specific telemetry, including prompts and completions, to reconstruct execution paths and debug AI workflows.
• AI Evaluation Frameworks - Provides a comprehensive framework for automating the assessment of AI outputs and reasoning quality.
• Model Evaluation Metrics - Calculates quantitative performance and quality metrics for trained machine learning models.
• Performance Metrics - Computes statistical performance indicators and dataframes by comparing production data against reference datasets for drift detection.
• RAG Evaluation Frameworks - Assesses RAG system performance using specialized metrics for groundedness, faithfulness, and retrieval relevance.
• RAG Performance Metrics - Measures the relationship between queries, retrieved contexts, and generated outputs to ensure RAG faithfulness.
• Production Evaluation Strategies - Runs scheduled batch evaluations on live production traffic to monitor for drift and quality loss.
• Dataset Comparators - Calculates distribution shifts by performing statistical tests between baseline datasets and live production data.
• Data Quality Monitors - Monitors data health and quality metrics for tabular datasets to ensure model reliability in production.
• Data Drift Detectors - Identifies statistical shifts between reference and production datasets to detect data drift in ML models.
• Model Health Monitors - Provides monitoring for ML-specific metrics like data drift and prediction accuracy to ensure model health in production.
• LLM-As-A-Judge Scoring - Implements an LLM-as-a-judge mechanism that uses language models to score and classify outputs based on custom rubrics.
• LLM Evaluation - Measures the quality and correctness of language model responses using automated judges and specialized metrics.
• Automated Prompt Optimization - Iteratively refines model instructions and few-shot examples based on quantitative performance metrics.
• Custom Evaluation Judges - Allows the definition of specialized evaluation logic and rubrics using prompt templates to score text outputs.
• RAG Evaluation Dataset Generation - Generates question-and-answer pairs from knowledge sources to evaluate the effectiveness of retrieval augmented generation systems.
• Synthetic Data Generators - Creates structured synthetic datasets using language models to facilitate testing and cold-starting AI applications.
• Evaluation Datasets - Organizes and manages testing and production datasets to create structured benchmark cases for model evaluation.
• Evaluation Report Aggregators - Consolidates individual AI assessment results into visual dashboards and comprehensive performance reports.
• Evaluation Visualizers - Provides dashboards for exploring and comparing AI evaluation datasets and performance metrics.
• Performance Trend Visualizations - Stores and visualizes evaluation results over time to compare changes across dataset versions and prompts.
• Evaluation Threshold Gates - Applies threshold-based decision logic to determine pass/fail status for AI evaluation scores.
• Prompt Optimizers - Provides tools for iteratively refining and testing prompts through systematic evaluation and comparison of model responses.
• Model Prediction Evaluation - Compares model predictions against ground truth labels to calculate accuracy for classification and regression tasks.
• Data Quality Profilers - Analyzes tabular datasets for missing values and descriptive statistics to ensure input data integrity.
• Metric Preset Templates - Bundles groups of related metrics into preset templates to standardize the analysis of tabular and generative data.
• CI/CD Pipeline Integrations - Integrates automated output testing into CI/CD pipelines to validate model quality and prevent performance regressions.
• Adversarial Safety Tests - Runs scenario-based risk tests with adversarial inputs to identify vulnerabilities regarding forbidden topics and brand safety.
• Adversarial Input Generation - Generates adversarial inputs and edge-case scenarios to perform safety evaluations and brand risk stress-testing on AI models.
• Alert Thresholds - Triggers automated notifications when AI quality scores cross predefined safety or performance thresholds.
• Performance Reporting - Transforms drift scores and metrics into standalone visual dashboards and reports for performance analysis.
• Performance Trend Analysis - Analyzes performance patterns and system health trends across different datasets via a monitoring dashboard.
• Real-Time Monitoring Dashboards - Ships real-time monitoring dashboards that track evaluation results and trigger alerts on performance violations.
• Testing & Quality Assurance - Executes quality checks and triggers alerts when AI performance falls below predefined thresholds.
• Automated Agent Quality Assurance - Integrates automated evaluation suites and regression tests into CI/CD pipelines to validate AI model updates.
• AI Regression Testing Suites - Automates regression testing for LLM behavior against quality standards within CI/CD pipelines.
• Regression Testing Suites - Groups evaluations into conditional suites to track system stability and prevent regressions over time.
• Application Development - Framework for monitoring and testing ML and LLM systems.
• General Machine Learning - Tool for analyzing ML models during validation and production.
• LLM Observability and Evaluation - Framework for evaluating and monitoring ML and LLM systems.
• Machine Learning Operations - Tool for analyzing and monitoring data and model drift.
• Model Evaluation and Benchmarking - Framework for evaluating, testing, and monitoring ML and LLM systems.
• Observability And Monitoring - Observability framework for tracking machine learning and language models.
• Data Validation - Monitors and evaluates machine learning models in production.
• Observability and Evaluation - Evaluation and monitoring framework for ML and LLM systems.

Star history