NVIDIA/NeMo-Guardrails
NeMo Guardrails
NeMo-Guardrails is a toolkit for adding programmable safety constraints and dialogue boundaries to large language model conversational systems. It functions as security middleware that intercepts inputs and outputs to block prompt injections, jailbreaks, and sensitive data leaks, while providing a conversational dialogue manager to define structured interaction flows through configuration files.
The framework includes a hallucination filter to screen model outputs for factual accuracy and a specialized modeling language for defining conversational flows and constraints. It provides capabilities for conversational dialogue steering to keep assistants on topic and uses safety moderation to block prohibited content.
The system covers broader capability areas including vulnerability testing and safety evaluation tooling to scan for weaknesses. It also provides observability through request tracing, retrieved context validation to filter sensitive information, and secure tool execution for agentic workflows.
The project can be deployed as a standalone HTTP server or via containerized microservices to provide protected chat completions to external clients.
Features
- LLM Guardrail Frameworks - Provides a comprehensive framework for enforcing safety and structural constraints on LLM inputs and outputs.
- Dialogue Flow Management - Provides the ability to define structured interaction flows and constraints using configuration files.
- Declarative Dialogue Modeling - Provides a specialized modeling language for defining conversational flows and constraints through declarative rules.
- Hallucination Detection - Includes mechanisms for identifying and scoring AI-generated content against retrieved context to prevent hallucinations.
- Safety Logic Integration - Enables the injection of safety logic into agentic workflows to maintain consistent standards across execution pipelines.
- Conversational Dialogue Systems - Controls the flow of AI conversations to ensure the assistant follows specific interaction scripts.
- Programmable Dialogue Constraints - Allows the definition of interaction flows and safety checks using a programmable modeling language.
- Dialogue Flow Orchestration - Uses a modeling language to map conversational paths and keep assistants on topic.
- Prompt Injection Detectors - Protects models from jailbreak attempts and malicious instructions using input inspection.
- Input Quality Filters - Inspects and modifies user prompts to mask sensitive data and block malicious content before they reach the model.
- LLM Safety Enforcers - Filters user inputs and model outputs to block jailbreaks, PII, and prohibited content.
- Output Guardrails - Validates and filters responses generated by language models against safety and accuracy criteria.
- LLM Security - Implements a security layer to identify and mitigate vulnerabilities specific to LLMs, such as prompt injection.
- Input and Output Guardrails - Intercepts and filters data flowing into and out of the model to ensure safety and integrity.
- Guardrail Execution Traces - Records each guardrail's action and content modifications as observability spans for monitoring and debugging.
- Guardrail Reliability Evaluation - Ships a dedicated tool for quantifying the accuracy and reliability of moderation and hallucination filters.
- Model-Driven Action Triggering - Implements mechanisms that allow the dialogue manager to execute custom Python functions and external tools.
- Context Retrieval Filtering - Filters retrieved data chunks against safety policies to remove sensitive information before prompting the model.
- Middleware-Based Request Pipelines - Implements a pipeline of middleware layers that intercept and modify messages before they reach the model.
- AI Agent Security - Applies validation and safety guardrails to the tools and multi-step pipelines used by AI agents.
- Command Execution Guardrails - Validates model outputs to ensure only safe and authorized system commands are executed via custom actions.
- LLM Security Scanning - Evaluates the system against common attack vectors to determine the effectiveness of existing security constraints.
- LLM Judge Accuracy Validators - Measures response reliability by validating outputs against expected results and safety guidelines.
- LLM Evaluation - Evaluates the effectiveness of security policies by scanning for weaknesses and running safety evaluations.
- AI Security and Red Teaming - Toolkit for adding programmable guardrails to LLM applications.
- Guardrails and AI Safety - Listed in the “Guardrails and AI Safety” section of the The Incredible Pytorch awesome list.
- Privacy and Safety - Toolkit for adding programmable safety guardrails to LLMs.
- Red Teaming and Security - Programmable guardrails for controlling conversational system behavior.
- Safety and Security - Toolkit for programmable conversational guardrails.
- Security & Privacy - Programmable toolkit for adding safety guardrails to conversational applications.
