NVIDIA/garak
Garak
Garak is an AI model evaluation tool and vulnerability scanner designed for red teaming large language models and auditing the security of retrieval-augmented generation pipelines. It identifies behavioral weaknesses, such as jailbreaks, hallucinations, and data leakage, by simulating adversarial attacks and executing automated testing vectors.
The framework utilizes an adaptive probing loop where prompts can react to previous model behavior and be modified in flight via middleware. To ensure consistent analysis, it employs a provider-agnostic interface to interact with various model APIs and maps all detected vulnerabilities to recognized industry security taxonomies.
The system provides capabilities for behavioral weakness detection, adversarial prompt testing, and the generation of detailed audit reports. It is built with a plugin-based architecture that allows for the extension of scanner capabilities through custom probes, detectors, and data generators.
Features
- LLM Vulnerability Scanners - Provides an automated scanner to identify prompt injection and data leakage vulnerabilities in large language models.
- Adversarial Testing - Executes dynamic probes to find specific inputs that trigger unintended or harmful model responses.
- Safety Auditing Reports - Generates detailed audit reports and maps model failures to industry taxonomies for safety compliance.
- Behavioral Weakness Detectors - Implements detectors to identify model failures like jailbreaks and misinformation by analyzing output behavior.
- Model Evaluation Tools - Measures model reliability by mapping vulnerability findings to recognized industry security taxonomies.
- Model Provider Integrations - Provides connectors for cloud-hosted and local large language models to enable AI-driven vulnerability probing.
- Model Red-Teaming - Simulates adversarial attacks to identify behavioral weaknesses, jailbreaks, and misinformation risks in AI models.
- Provider-Agnostic Model Interfaces - Abstracts various LLM APIs and local model files into a common interface for consistent testing.
- Retrieval-Augmented Generation Security - Evaluates RAG pipelines for indirect prompt injection and other vulnerabilities in the data retrieval flow.
- Adaptive Probing Loops - Executes reactive probing loops that adapt inputs based on previous model behavior to uncover weaknesses.
- Adversarial Probing Loops - Implements an adaptive probing loop that generates prompts reacting to previous model behavior to uncover weaknesses.
- LLM - Probes large language models for hallucinations, data leakage, and prompt injection using automated testing vectors.
- RAG Pipeline Scanning - Audits retrieval-augmented generation pipelines specifically for vulnerabilities like indirect prompt injection.
- Model Response Validation Wrappers - Evaluates model outputs using specialized validation layers to identify failures like jailbreaks.
- Prompt Parsers - Transforms and paraphrases input prompts in flight using a pipeline of modifiers before model delivery.
- Prompt Processing - Provides middleware to transform and paraphrase prompts in flight before they reach the model.
- Framework Capability Extensions - Enables implementation of custom probes and detectors through a plugin-based framework extension.
- Adversarial Data Generators - Allows the creation of specific input data types for probing models via custom generator classes.
- Vulnerability Mapping - Categorizes security risks by grouping vulnerability results according to recognized industry standards.
- Custom Security Scan Extensions - Features a plugin-based architecture for adding custom probes, detectors, and testing vectors.
- Security Report Generation - Produces detailed records of model parameters, prompts, and responses to ensure testing transparency.
- Vulnerability Category Mapping - Groups individual vulnerability hits into standardized security categories for high-level risk reporting.
- Plugin-Based Architectures - Provides a plugin-based architecture for adding custom probes, detectors, and data generators.
- AI Red Teaming - Vulnerability scanner for probing LLM weaknesses and jailbreaks.
- AI Security and Red Teaming - Red-teaming kit for probing LLMs for hallucinations and data leakage.
- AI Security Frameworks - Comprehensive vulnerability scanner for large language models.
- Red Teaming and Security - Vulnerability scanner for detecting LLM hallucinations and jailbreaks.
- Safety and Security - Vulnerability scanner for identifying model weaknesses.
- Security And Privacy - Listed in the “Security And Privacy” section of the Llm Course awesome list.
- Security and Red Teaming - Vulnerability scanner for detecting prompt injection and jailbreak risks.
- Security and Sandboxing - Vulnerability scanner for language models.
