facebookresearch/PurpleLlama
PurpleLlama
PurpleLlama is a collection of security toolsets and frameworks designed to audit large language model vulnerabilities and implement runtime input-output guardrails. It provides a security evaluation framework and benchmark suite to quantify risks associated with prompt injections and the generation of malicious code.
The project includes a content moderator and input-output filters that use a standardized taxonomy to identify and block harmful content, jailbreaking attempts, and insecure commands. It also features capabilities for sensitive document classification to prevent the unauthorized exposure of private information.
The system covers a broad security surface including multi-language safety classification, execution-time code scanning for vulnerabilities, and cybersecurity risk benchmarking based on industry-standard frameworks.
Features
- Content Moderation - Ships a safety model that classifies inputs and outputs to block harmful content and jailbreaking attempts.
- Security Evaluation Suites - Provides a collection of standardized metrics and tests to assess the susceptibility of language models to security vulnerabilities.
- Input Filters - Implements filters that analyze incoming prompts for jailbreak attempts and sensitive data before they reach the model.
- LLM Evaluation Frameworks - Offers a set of tools and benchmarks for quantifying the risk of prompt injections and malicious code generation.
- AI Risk Assessments - Quantifies the propensity to generate malicious code or succumb to prompt injections using industry-standard security frameworks.
- AI Content Filters - Detects and blocks unsafe or prohibited inputs and outputs to ensure responses adhere to safety guidelines.
- Insecure API Detection - Scans generated code for dangerous functions and insecure programming patterns to prevent malicious command execution.
- AI Output Command Filtering - Performs security scans of generated code during execution to prevent insecure suggestions and malicious command execution.
- LLM Prompt Injection Prevention - Identifies and blocks prompt injection and jailbreaking attempts to maintain the security and integrity of the application.
- LLM Security Benchmarks - Provides a benchmark suite to quantify risks associated with prompt injections and malicious code generation.
- LLM Vulnerability Assessments - Provides a comprehensive framework and benchmark suite to quantify the susceptibility of language models to security vulnerabilities.
- Input and Output Guardrails - Provides a guardrail system that scans prompts and generated code to prevent insecure commands and data exposure.
- LLM Security - Tests large language models for vulnerabilities to prompt injections and the generation of malicious code.
- AI Generated Code Scanning - Inspects generated code fragments for security vulnerabilities and malicious commands prior to execution.
- Safety Classifications - Identifies violating content across various languages using a unified taxonomy for consistent policy enforcement.
- Data Sensitivity Classifications - Groups documents based on data sensitivity levels to prevent the unauthorized exposure of private information.
- AI Security Frameworks - Open ecosystem for building safe and secure AI models.
