# promptfoo/promptfoo

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/promptfoo-promptfoo).**

10,529 stars · 928 forks · TypeScript · mit

## Links

- GitHub: https://github.com/promptfoo/promptfoo
- Homepage: https://promptfoo.dev
- awesome-repositories: https://awesome-repositories.com/repository/promptfoo-promptfoo.md

## Topics

`ci` `ci-cd` `cicd` `evaluation` `evaluation-framework` `llm` `llm-eval` `llm-evaluation` `llm-evaluation-framework` `llmops` `pentesting` `prompt-engineering` `prompt-testing` `prompts` `rag` `red-teaming` `testing` `vulnerability-scanners`

## Description

Promptfoo is an evaluation framework designed for testing, benchmarking, and red-teaming language models and agentic workflows. It provides a unified environment to run prompts against multiple providers, allowing developers to systematically validate model outputs against objective assertions, semantic similarity metrics, and custom grading rubrics.

The platform distinguishes itself through a provider-agnostic execution layer and a stateful orchestrator capable of simulating multi-turn conversations and complex tool-use trajectories. It includes a dedicated adversarial mutation pipeline that automates security vulnerability scanning, enabling teams to probe for jailbreaks, prompt injections, and safety policy violations using systematic attack strategies.

Beyond core testing, the project supports comprehensive quality assurance through retrieval-augmented generation assessment, synthetic dataset generation, and prompt performance optimization. It offers extensive extensibility through a plugin-based architecture, allowing for custom logic, Python-based testing extensions, and integration with external version control and observability platforms.

The system utilizes a declarative configuration schema to manage test cases and environment settings, supporting both self-hosted and managed infrastructure deployments. Results are consolidated into structured reports with interactive visualizations to facilitate collaborative review and integration into continuous integration pipelines.

## Tags

### Testing & Quality Assurance

- [LLM Evaluation](https://awesome-repositories.com/f/testing-quality-assurance/model-testing/llm-evaluation.md) — Provides a comprehensive framework for testing, benchmarking, and red-teaming language models across multiple providers.
- [Automated Assertion Validators](https://awesome-repositories.com/f/testing-quality-assurance/validation-verification/input-validation/agent-input-and-output-validators/automated-assertion-validators.md) — Validates language model outputs against deterministic rules, semantic similarity metrics, and custom scripts to verify quality and safety.
- [Test Case Definitions](https://awesome-repositories.com/f/testing-quality-assurance/software-testing/testing-frameworks/test-frameworks/assertions-and-validation/test-case-definitions.md) — Provides structured test case definitions to validate model outputs against expected outcomes and assertions. ([source](https://www.promptfoo.dev/docs/category/configuration/))
- [Test Utilities & Assertions](https://awesome-repositories.com/f/testing-quality-assurance/general-testing-utilities/test-utilities-assertions.md) — Provides objective and subjective criteria for validating model outputs via assertions and rubrics. ([source](https://www.promptfoo.dev/docs/usage/command-line/))
- [Test Report Aggregators](https://awesome-repositories.com/f/testing-quality-assurance/general-testing-utilities/test-utilities-assertions/coverage-diagnostics-reporting/test-report-aggregators.md) — Consolidates performance metrics and security findings into structured reports for visualization and integration with development pipelines.
- [Automated Test Runners](https://awesome-repositories.com/f/testing-quality-assurance/software-testing/test-execution-orchestration/automated-test-runners.md) — Orchestrates automated evaluation workflows within CI pipelines to track regressions and report results. ([source](https://www.promptfoo.dev/docs/category/integrations/))
- [Test Report Servers](https://awesome-repositories.com/f/testing-quality-assurance/general-testing-utilities/test-report-servers.md) — Distributes test findings and performance metrics to team members through cloud-based platforms or self-hosted infrastructure for collaborative review. ([source](https://www.promptfoo.dev/docs/category/usage/))
- [Test Assertion Extensions](https://awesome-repositories.com/f/testing-quality-assurance/general-testing-utilities/test-utilities-assertions/assertion-extensions-modes/test-assertion-extensions.md) — Allows writing custom providers, assertions, and test generators using Python to integrate with external frameworks and libraries. ([source](https://www.promptfoo.dev/docs/category/integrations/))
- [Test Case Generators](https://awesome-repositories.com/f/testing-quality-assurance/software-testing/test-execution-orchestration/test-case-generators.md) — Automatically generates diverse test cases and personas to expand evaluation coverage. ([source](https://www.promptfoo.dev/docs/usage/command-line/))

### Artificial Intelligence & ML

- [Prompt Engineering Toolkits](https://awesome-repositories.com/f/artificial-intelligence-ml/language-model-orchestration/prompt-engineering-toolkits.md) — Offers a toolkit for iteratively refining, comparing, and optimizing prompt templates and model configurations.
- [Adversarial Security Research](https://awesome-repositories.com/f/artificial-intelligence-ml/ai-security-and-governance/adversarial-security-research.md) — Automates security vulnerability scanning by generating and chaining malicious inputs to probe for jailbreaks and prompt injections.
- [LLM Provider Integrations](https://awesome-repositories.com/f/artificial-intelligence-ml/llm-provider-integrations.md) — Connects to various commercial and open-source model APIs to run comparative benchmarks and evaluations within a unified environment. ([source](https://www.promptfoo.dev/docs/intro/))
- [RAG Evaluation Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/rag-evaluation-frameworks.md) — Assesses the factual accuracy and relevance of retrieval-augmented generation pipelines by comparing responses against source data.
- [Stateful Agent Orchestration](https://awesome-repositories.com/f/artificial-intelligence-ml/stateful-agent-orchestration.md) — Manages multi-turn conversation histories and tool-use trajectories to simulate complex agentic workflows during automated testing.
- [Agent Evaluation Tools](https://awesome-repositories.com/f/artificial-intelligence-ml/agent-evaluation-tools.md) — Simulates multi-turn interactions and tool usage to verify agentic task execution. ([source](https://www.promptfoo.dev/docs/getting-started/))
- [Agent Framework Integrations](https://awesome-repositories.com/f/artificial-intelligence-ml/agent-framework-integrations.md) — Connects with orchestration libraries to test, trace, and evaluate multi-step workflows and complex agentic applications. ([source](https://www.promptfoo.dev/docs/guides/))
- [AI Model Integrations](https://awesome-repositories.com/f/artificial-intelligence-ml/ai-model-integrations.md) — Links to a wide range of hosted, local, and custom AI providers through a unified interface for consistent testing. ([source](https://www.promptfoo.dev/docs/providers/))
- [Automated Prompt Optimization](https://awesome-repositories.com/f/artificial-intelligence-ml/automated-prompt-optimization.md) — Iteratively refines prompts based on performance metrics to identify the most effective versions. ([source](https://www.promptfoo.dev/docs/usage/command-line/))
- [Validation Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/language-model-orchestration/retrieval-augmented-generation/rag-pipelines/validation-frameworks.md) — Assesses retrieval-augmented generation accuracy by comparing model responses against source data.
- [Hardware-Agnostic Inference Layers](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-inference-serving/inference-engines/hardware-agnostic-inference-layers.md) — Standardizes communication across diverse model APIs and local scripts to enable unified testing and comparative benchmarking.
- [Agent Tool Integrations](https://awesome-repositories.com/f/artificial-intelligence-ml/agentic-systems-frameworks/integration-deployment/agent-frameworks/tool-use-and-execution/agent-tool-integrations.md) — Provides mechanisms for connecting autonomous agents to external software tools and APIs to extend their functional capabilities during evaluations. ([source](https://www.promptfoo.dev/docs/providers/))
- [Model Comparison Interfaces](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-evaluation-analysis/machine-learning-evaluation/model-comparison-interfaces.md) — Provides side-by-side comparison of model versions and prompt templates to identify optimal configurations. ([source](https://www.promptfoo.dev/docs/guides/evaluate-rag/))
- [Provider Configurations](https://awesome-repositories.com/f/artificial-intelligence-ml/provider-configurations.md) — Stores and references API keys and base settings in a central environment while maintaining the ability to override parameters locally. ([source](https://www.promptfoo.dev/docs/providers/))
- [Tool Definition Adapters](https://awesome-repositories.com/f/artificial-intelligence-ml/agentic-systems-frameworks/integration-deployment/agent-frameworks/tool-definitions-and-registration/tool-definition-adapters.md) — Standardizes tool definitions across multiple model providers to ensure consistent behavior and reliable execution during automated evaluation cycles. ([source](https://www.promptfoo.dev/docs/category/configuration/))
- [Conversational Evaluation Suites](https://awesome-repositories.com/f/artificial-intelligence-ml/conversational-evaluation-suites.md) — Models multi-turn conversation histories to evaluate stateful dialogue flows. ([source](https://www.promptfoo.dev/docs/category/configuration/))
- [Evaluation Report Aggregators](https://awesome-repositories.com/f/artificial-intelligence-ml/evaluation-metrics/evaluation-report-aggregators.md) — Consolidates individual test session results into comprehensive performance reports for centralized tracking. ([source](https://www.promptfoo.dev/docs/providers/))
- [External Knowledge Integrators](https://awesome-repositories.com/f/artificial-intelligence-ml/external-service-integrations/external-knowledge-integrators.md) — Connects agents to external databases and APIs to inject domain-specific information for retrieval-augmented generation assessment. ([source](https://www.promptfoo.dev/docs/guides/prevent-llm-hallucinations/))
- [Custom Model Logic Interfaces](https://awesome-repositories.com/f/artificial-intelligence-ml/neural-network-implementations/lightweight-model-implementations/custom-model-logic-interfaces.md) — Allows defining bespoke provider behavior using scripts or local files to test proprietary models or unique workflows. ([source](https://www.promptfoo.dev/docs/providers/))
- [Prompt Management Systems](https://awesome-repositories.com/f/artificial-intelligence-ml/prompt-management-systems.md) — Connects to external version control and observability platforms to monitor, track, and optimize prompt performance. ([source](https://www.promptfoo.dev/docs/category/integrations/))
- [Token Bias Adjustments](https://awesome-repositories.com/f/artificial-intelligence-ml/text-generation-strategies/token-prediction/token-bias-adjustments.md) — Forces model responses to adhere to specific choices by applying logit bias to ensure structured and predictable outputs. ([source](https://www.promptfoo.dev/docs/guides/prevent-llm-hallucinations/))

### Security & Cryptography

- [Automated Prompt Testing](https://awesome-repositories.com/f/security-cryptography/security/ai-and-machine-learning/prompt-injection-testing/automated-prompt-testing.md) — Evaluation & Testing triggers systematic quality and performance tests for prompts automatically whenever code changes are pushed to a repository. ([source](https://www.promptfoo.dev/docs/integrations/github-action/))
- [Adversarial Red Teaming Toolkits](https://awesome-repositories.com/f/security-cryptography/security/offensive-operations/vulnerability-research-analysis/analysis-discovery-tooling/adversarial-testing-resources/adversarial-red-teaming-toolkits.md) — Automates the detection of jailbreaks, prompt injections, and safety violations by running adversarial test cases against language models. ([source](https://www.promptfoo.dev/docs/red-team/quickstart/))
- [Automated Security Scanners](https://awesome-repositories.com/f/security-cryptography/vulnerability-assessment-testing/security-testing-auditing/security-testing-tools/reconnaissance-assessment-platforms/automated-security-scanners.md) — Automates adversarial testing and vulnerability detection for language models and agentic workflows.
- [Prompt Injection Testing](https://awesome-repositories.com/f/security-cryptography/security/ai-and-machine-learning/prompt-injection-testing.md) — Generates external content with hidden instructions to evaluate agent manipulation risks. ([source](https://www.promptfoo.dev/docs/red-team/strategies/))
- [Adversarial Test Automation](https://awesome-repositories.com/f/security-cryptography/security/offensive-operations/vulnerability-research-analysis/analysis-discovery-tooling/adversarial-testing-resources/adversarial-red-teaming-toolkits/adversarial-test-automation.md) — Executes modular test suites that generate malicious payloads to identify security and compliance risks in language models. ([source](https://www.promptfoo.dev/docs/red-team/plugins/))
- [Content Guardrails](https://awesome-repositories.com/f/security-cryptography/content-guardrails.md) — Validates the effectiveness of safety filters and moderation layers by simulating adversarial inputs. ([source](https://www.promptfoo.dev/docs/red-team/guides/))
- [AI Governance Policies](https://awesome-repositories.com/f/security-cryptography/governance-policy-frameworks/compliance-governance/security-governance/security-policy-configurations/ai-governance-policies.md) — Creates bespoke testing plugins to enforce organization-specific behavioral standards and AI governance policies. ([source](https://www.promptfoo.dev/docs/red-team/plugins/))
- [Adversarial Input Transformers](https://awesome-repositories.com/f/security-cryptography/security/offensive-operations/vulnerability-research-analysis/analysis-discovery-tooling/adversarial-testing-resources/adversarial-red-teaming-toolkits/adversarial-input-transformers.md) — Transforms test inputs using techniques like obfuscation to bypass content filters and security controls. ([source](https://www.promptfoo.dev/docs/red-team/configuration/))
- [Model Vulnerability Scanners](https://awesome-repositories.com/f/security-cryptography/security/offensive-operations/vulnerability-research-analysis/analysis-discovery-tooling/adversarial-testing-resources/adversarial-red-teaming-toolkits/model-vulnerability-scanners.md) — Provides automated scanning of model files to detect security risks and architectural vulnerabilities before deployment. ([source](https://www.promptfoo.dev/docs/model-audit/))

### Development Tools & Productivity

- [AI Agent Benchmarks](https://awesome-repositories.com/f/development-tools-productivity/debugging-profiling-testing/ai-agent-benchmarks.md) — Simulates complex multi-turn interactions and tool usage to verify agentic workflow reliability.
- [Data-Driven Testing](https://awesome-repositories.com/f/development-tools-productivity/data-driven-testing.md) — Imports and exports test cases and evaluation results using external spreadsheet or document management systems for collaborative data handling. ([source](https://www.promptfoo.dev/docs/category/integrations/))

### Software Engineering & Architecture

- [Evaluation Templates](https://awesome-repositories.com/f/software-engineering-architecture/declarative-configuration-schemas/evaluation-templates.md) — Structures test cases, prompt templates, and evaluation criteria into portable files for consistent execution across environments.
- [Extensible Plugin Architectures](https://awesome-repositories.com/f/software-engineering-architecture/extensible-plugin-architectures.md) — Supports modular extension by allowing developers to inject custom logic for providers, grading rubrics, and attack strategies.

### System Administration & Monitoring

- [Model Observability Suites](https://awesome-repositories.com/f/system-administration-monitoring/model-observability-suites.md) — Monitoring & Observability calculates perplexity scores for model outputs to quantify prediction certainty and identify potential hallucination risks based on configurable thresholds. ([source](https://www.promptfoo.dev/docs/guides/prevent-llm-hallucinations/))
- [Performance Visualization](https://awesome-repositories.com/f/system-administration-monitoring/monitoring-and-observability/observability-platforms/metric-performance-monitors/performance-visualization.md) — Provides interactive side-by-side visualization of model outputs to facilitate comparison and manual rating. ([source](https://www.promptfoo.dev/docs/category/usage/))
- [Language Model Metrics](https://awesome-repositories.com/f/system-administration-monitoring/monitoring-and-observability/ai-agent-observability/language-model-metrics.md) — Uses model-based grading and embedding comparisons to assess factual accuracy and faithfulness. ([source](https://www.promptfoo.dev/docs/guides/evaluate-rag/))
- [Evaluation Grading Configurations](https://awesome-repositories.com/f/system-administration-monitoring/observability-configurations/evaluation-grading-configurations.md) — Monitoring & Observability customizes the grading model, scoring weights, and evaluation prompts to tailor accuracy assessments to specific domain requirements. ([source](https://www.promptfoo.dev/docs/guides/factuality-eval/))
- [Composite Metric Calculators](https://awesome-repositories.com/f/system-administration-monitoring/monitoring-and-observability/observability-platforms/metric-performance-monitors/composite-metric-calculators.md) — Monitoring & Observability calculates composite scores from individual assertion results using mathematical expressions or scripts to generate custom performance indicators. ([source](https://www.promptfoo.dev/docs/configuration/expected-outputs/))

### DevOps & Infrastructure

- [Continuous Integration Quality Gates](https://awesome-repositories.com/f/devops-infrastructure/cicd-pipeline-automation/cicd-pipeline-management/continuous-integration-tools/continuous-integration-quality-gates.md) — Integrates automated quality gates into CI pipelines to enforce performance standards and prevent regressions.
- [Managed Infrastructure Support](https://awesome-repositories.com/f/devops-infrastructure/deployment-management/self-hosted-infrastructure-management/managed-infrastructure-support.md) — Provides a fully-managed service for hosting evaluation environments to eliminate infrastructure maintenance. ([source](https://www.promptfoo.dev/docs/enterprise/))
- [Self-Hosted Infrastructure](https://awesome-repositories.com/f/devops-infrastructure/self-hosted-infrastructure.md) — Supports self-hosted deployment within private networks to ensure data sovereignty and control. ([source](https://www.promptfoo.dev/docs/enterprise/))

### Data & Databases

- [Response Caching](https://awesome-repositories.com/f/data-databases/response-caching.md) — Stores model call results locally to reduce latency and costs during repeated test executions. ([source](https://www.promptfoo.dev/docs/configuration/caching/))