Projects sharing features with LLMFuzzer

PyRIT is an AI vulnerability assessment tool and security scanner designed to detect risks in large language model applications. It functions as a generative AI red teaming framework used to simulate adversarial attacks and identify weaknesses in system guardrails. The tool automates AI risk assessment by scanning generative AI components for security vulnerabilities. It utilizes automated testing and analysis to identify security gaps and prevent potential exploits through a consistent, repeatable process. The system incorporates asynchronous model orchestration to compare security postures

Promptfoo is an evaluation framework designed for testing, benchmarking, and red-teaming language models and agentic workflows. It provides a unified environment to run prompts against multiple providers, allowing developers to systematically validate model outputs against objective assertions, semantic similarity metrics, and custom grading rubrics. The platform distinguishes itself through a provider-agnostic execution layer and a stateful orchestrator capable of simulating multi-turn conversations and complex tool-use trajectories. It includes a dedicated adversarial mutation pipeline that

A collection of real world AI/ML exploits for responsibly disclosed vulnerabilities

Superagent is an AI safety platform that protects applications from prompt injections, data leaks, and harmful outputs through built-in guardrails. It functions as a prompt injection detection system, data redaction tool, and red team testing tool, automatically removing personally identifiable information and protected health data from AI inputs and outputs while scanning image uploads with vision AI to detect visual prompt injection attacks before processing. The platform routes every prompt through a sequential pipeline of safety checks including injection detection, data redaction, and co

Learn_Prompting is an educational project focused on prompt engineering, providing the principles and techniques required to craft effective inputs and improve the quality of generative AI outputs. The project covers advanced prompting strategies to enhance reasoning, reliability, and output quality. This includes techniques for task decomposition, chain-of-thought reasoning, and the use of few-shot and zero-shot guidance. It also addresses model security through the study of prompt hacking, vulnerability analysis, and privacy auditing to prevent sensitive data leaks. The scope extends to th

This project is an educational and research platform designed to simulate security vulnerabilities within AI-integrated systems and Model Context Protocol implementations. It provides a controlled environment where users can practice identifying and mitigating common attack vectors, such as prompt injection and unauthorized code execution, by interacting with intentionally insecure tools and protocol configurations. The platform distinguishes itself by offering a dedicated laboratory for auditing Model Context Protocol integrations. It exposes server-side functions as discoverable tools and p

A curation of awesome tools, documents and projects about LLM Security.

⚡ Security scanner for LLM prompts ⚡

Multi-engine security scanner for AI agents, MCP servers & plugins — 13 engines, one report.

IATelligence is a Python script that will extract the IAT of a PE file and request GPT to get more information about the API and the ATT&CK matrix related

We present a new class of vulnerabilities and impacts stemming from "indirect prompt injection" affecting language models integrated with applications. Our demos currently span GPT-4 (Bing and synthetic apps) using ChatML, GPT-3 & LangChain based apps in addition to proof-of-concepts for attacks…

An Intelligent Automated Binary Vulnerability Analysis Tool

A curated list of useful resources that cover Offensive AI.

k8sgpt is a suite of Kubernetes-focused tools designed for AI-powered debugging, cluster diagnostics, and self-healing. It functions as an automated analyzer and debugger that uses large language models to explain cluster errors, suggest remediation steps, and identify resource failures. The project distinguishes itself through an extensible analysis framework that supports custom diagnostic plugins and a Model Context Protocol server, which exposes cluster diagnostics as tools for AI assistants. It includes a self-healing agent capable of automatically generating and applying fixes for detec

Shannon is an integrated security platform designed for autonomous penetration testing, static and dynamic analysis, and automated vulnerability remediation within self-hosted, private infrastructure. It functions as a unified security suite that orchestrates the entire lifecycle of vulnerability management, from initial discovery and reachability prioritization to the generation and verification of code-level patches. The platform distinguishes itself through its agentic approach to security, deploying autonomous agents to execute both black-box and white-box exploits against running applica

Netron is a visualizer for neural network and machine learning models. It provides a graphical interface that renders model architectures as interactive node-link diagrams, allowing users to inspect internal layers, tensors, and metadata. By performing static analysis, the tool enables the examination of model definitions without executing the underlying machine learning code. The software distinguishes itself through a schema-driven parsing engine that translates diverse proprietary model formats into a unified internal graph structure. This approach ensures interoperability, allowing users

Garak is an AI model evaluation tool and vulnerability scanner designed for red teaming large language models and auditing the security of retrieval-augmented generation pipelines. It identifies behavioral weaknesses, such as jailbreaks, hallucinations, and data leakage, by simulating adversarial attacks and executing automated testing vectors. The framework utilizes an adaptive probing loop where prompts can react to previous model behavior and be modified in flight via middleware. To ensure consistent analysis, it employs a provider-agnostic interface to interact with various model APIs and

NeMo-Guardrails is a toolkit for adding programmable safety constraints and dialogue boundaries to large language model conversational systems. It functions as security middleware that intercepts inputs and outputs to block prompt injections, jailbreaks, and sensitive data leaks, while providing a conversational dialogue manager to define structured interaction flows through configuration files. The framework includes a hallucination filter to screen model outputs for factual accuracy and a specialized modeling language for defining conversational flows and constraints. It provides capabiliti

SkillSpector is a security scanner designed to detect vulnerabilities and malicious patterns in AI agent plugins and extensions before they are installed. It functions as a runtime guardrail that calculates numeric risk scores and assigns severity labels to provide installation recommendations or block risky external extensions. The project distinguishes itself by using language models to perform semantic code analysis, evaluating code intent and context to reduce false positives. It also employs fingerprint-based issue suppression to track and ignore previously accepted risks across repeated