confident-ai/deepeval

Features

• AI Regression Testing Suites - Provides a suite for executing automated test cycles and validating model behavior against defined quality standards.
• LLM Evaluation - Uses secondary language models to evaluate and quantify the quality of outputs from primary models against predefined criteria.
• Automated Assertion Validators - Provides programmatic assertion-driven validation to ensure model outputs meet defined quality standards during development.
• LLM Observability - Traces and debugs complex multi-step agent execution workflows to identify performance bottlenecks and failures.
• Deployment Quality Gates - Blocks software deployments automatically when model outputs fail to meet performance requirements during continuous integration.
• Adversarial Red Teaming Toolkits - Performs security and safety red teaming to identify vulnerabilities and harmful behaviors in language models before deployment.
• Model Observability Suites - Offers comprehensive observability suites for tracing, monitoring, and evaluating the quality of language model outputs.
• Agent Testing Suites - Validates the reliability and behavior of autonomous agents by simulating complex workflows and inspecting multi-step execution traces.
• Workflow Performance Scorers - Quantifies the quality of AI outputs and agent workflows using automated scoring to ensure consistent performance.
• CI/CD Regression Analyzers - Executes automated test suites within continuous integration environments to detect performance regressions before deployment.
• Automated Test Execution - Executes repeatable automated regression test suites to verify model behavior and identify performance drops before production deployment.
• CI/CD Pipeline Integrations - Automates quality gates for AI applications to prevent performance regressions and security vulnerabilities from reaching production.
• Agent Execution Tracing - Captures internal component interactions and tool calls to provide visibility into multi-step agent workflows.
• Execution Observability - Captures internal model processes and execution history to debug complex agent workflows and identify performance bottlenecks.
• AI and Agent Observability - Evaluates the effectiveness of complex retrieval pipelines and conversational agents by applying specialized metrics.
• Assertion and Validation Utilities - Provides programmatic assertion utilities to verify model output quality against defined performance thresholds.
• Evaluation Datasets - Manages structured evaluation datasets to ensure consistent benchmarking across model versions and prompt iterations.
• Synthetic Data Generation - Generates synthetic datasets using language models to simulate edge cases and improve evaluation robustness.
• Synthetic Data Generators - Creates artificial test cases by leveraging language models to simulate diverse edge scenarios for robust system evaluation.
• AI Observability and Evaluation - Framework for testing and evaluating LLM applications.
• AI Red Teaming - Framework for evaluating large language model systems.
• Evaluation and Observability - Unit testing framework for LLM outputs.
• Evaluation Frameworks - Framework for unit testing and evaluating language model outputs.
• Large Language Models - Evaluation framework for LLM applications.
• LLM Evaluation Frameworks - Framework for unit testing and evaluating LLM outputs.
• LLM Evaluation Tools - Framework for evaluating RAG, agents, and conversations with CI/CD integration.
• Model Evaluation - Open-source framework for testing language model systems.
• Model Evaluation and Benchmarking - Simple framework for evaluating LLM-based applications.
• Reliability and Debugging - Pytest-style unit testing framework for LLMs. Metrics for RAG, agents, hallucination, summarization, and custom criteria.
• Retrieval Augmented Generation - Listed in the “Retrieval Augmented Generation” section of the Llm Course awesome list.
• Evaluation Frameworks - Unit testing framework for LLM applications using various NLP metrics.
• Dataset Management Tools - Manages evaluation test datasets to ensure consistent and repeatable performance benchmarking across prompts and model versions.
• Workflow Debugging - Analyzes internal execution steps and component interactions to troubleshoot failures within complex retrieval and conversational systems.
• Coding Agent Integrations - Integrates evaluation tools into coding agents to automatically generate, run, and refine test cases during development.

Star history