NVIDIA/garak
Garak
Garak is an AI model evaluation tool and vulnerability scanner designed for red teaming large language models and auditing the security of retrieval-augmented generation pipelines. It identifies behavioral weaknesses, such as jailbreaks, hallucinations, and data leakage, by simulating adversarial attacks and executing automated testing vectors.
The framework utilizes an adaptive probing loop where prompts can react to previous model behavior and be modified in flight via middleware. To ensure consistent analysis, it employs a provider-agnostic interface to interact with various model APIs and maps all detected vulnerabilities to recognized industry security taxonomies.
The system provides capabilities for behavioral weakness detection, adversarial prompt testing, and the generation of detailed audit reports. It is built with a plugin-based architecture that allows for the extension of scanner capabilities through custom probes, detectors, and data generators.
Features
- LLM Vulnerability Scanners - Provides an automated scanner to identify prompt injection and data leakage vulnerabilities in large language models.
- Adversarial Testing - Executes dynamic probes to find specific inputs that trigger unintended or harmful model responses.
- Safety Auditing Reports - Generates detailed audit reports and maps model failures to industry taxonomies for safety compliance.
- Behavioral Weakness Detectors - Implements detectors to identify model failures like jailbreaks and misinformation by analyzing output behavior.
