30 open-source projects similar to greshake/llm-security, ranked by how many features they have in common. Compare stars, activity and what each one does to find the best Llm Security alternative.
The Adversarial Robustness Toolbox (ART) is an open-source library that provides a unified framework for evaluating, defending, and certifying machine learning models against adversarial threats. It wraps models from any framework behind a common estimator interface, enabling composable pipelines for attack generation, defense application, robustness certification, and privacy auditing across evasion, poisoning, and extraction threats. The library distinguishes itself by covering the full adversarial ML security lifecycle within a single toolkit. It supports gradient-based adversarial example
PromptInject is a framework that assembles prompts in a modular fashion to provide a quantitative analysis of the robustness of LLMs to adversarial prompt attacks. 🏆 Best Paper Awards @ NeurIPS ML Safety Workshop 2022
PyRIT is an AI vulnerability assessment tool and security scanner designed to detect risks in large language model applications. It functions as a generative AI red teaming framework used to simulate adversarial attacks and identify weaknesses in system guardrails. The tool automates AI risk assessment by scanning generative AI components for security vulnerabilities. It utilizes automated testing and analysis to identify security gaps and prevent potential exploits through a consistent, repeatable process. The system incorporates asynchronous model orchestration to compare security postures
In this paper, we conduct the first work to propose the concept of indirect jailbreak and achieve Retrieval-Augmented Generation (RAG) via LangChain. Building on this, we further design a novel method of indirect jailbreak attack, termed Poisoned-LangChain (PLC), which leverages a poisoned…
Cleverhans is an adversarial machine learning library and toolkit designed to generate adversarial examples, incorporate them into training loops, and benchmark the resilience of machine learning models. It provides a gradient-based attack framework for constructing both white-box and black-box attacks to identify model misclassifications. The project includes capabilities for model robustness benchmarking, allowing users to evaluate and verify how models resist evasion attacks and malicious input perturbations. It also facilitates adversarial training to increase a model's resistance to pert
A curation of awesome tools, documents and projects about LLM Security.
The respositiy is public package of the paper titled "Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues" submitted to ACL 2024.
Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models
This repository is for our new work: "BadReasoner: Planting Tunable Overthinking Backdoors into Large Reasoning Models for Fun or Profit", feel free to propose your issues!! 😎
This is the source code for the ICLR 2024 paper "An LLM can Fool Itself: A Prompt-Based Adversarial Attack", Xilie Xu (NUS), Keyi Kong (SDU), Ning Liu (SDU), Lizhen Cui (SDU), Di Wang (KAUST), Jingfeng Zhang (University of Auckland/RIKEN-AIP), Mohan Kankanhalli (NUS).…
Paper Link: https://arxiv.org/pdf/2304.05197.pdf
This repository contains the code implementation for the paper CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models.
A curated list of useful resources that cover Offensive AI.
Garak is a suite of tools for measuring AI reliability, scanning for vulnerabilities, and automating security assessments through adaptive probing. It functions as a generative AI vulnerability scanner and evaluation tool designed to identify security gaps, hallucinations, and failure modes in language models. The framework provides a toolkit for red-teaming and safety assessments, utilizing a structured system of probes and detectors to calculate failure rates. It specifically scans for risks such as data leakage and prompt injection by recording model responses to adversarial inputs. The p
This is the replication package for the paper MASTERKEY: Automated Jailbreaking of Large Language Model Chatbots.
Code for our NAACL 2024 Paper "Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking"
This project is no longer actively maintained. You are welcome to fork and continue its development on your own. Thank you for your interest and support.
Official implementation for "Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy"
The official implementation of our NAACL 2024 paper "A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily".
Promptfoo is an evaluation framework designed for testing, benchmarking, and red-teaming language models and agentic workflows. It provides a unified environment to run prompts against multiple providers, allowing developers to systematically validate model outputs against objective assertions, semantic similarity metrics, and custom grading rubrics. The platform distinguishes itself through a provider-agnostic execution layer and a stateful orchestrator capable of simulating multi-turn conversations and complex tool-use trajectories. It includes a dedicated adversarial mutation pipeline that
A list of curated resources for people interested in AI Red Teaming, Jailbreaking, and Prompt Injection
A collection of real world AI/ML exploits for responsibly disclosed vulnerabilities
This repository contains a PyTorch implementation for our paper Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation.
QROA, or Query-Response Optimization Attack, is an innovative and robust strategy designed to explore and exploit vulnerabilities in Large Language Models (LLMs) through black-box interactions. This method leverages optimized triggers embedded within benign-looking instructions to manipulate…
This repository contains code and data for "FUZZLLM: A Novel and Universal Fuzzing Framework for Discovering Jailbreak Vulnerabilities in LLMs" (accepted to ICASSP 2024). Our work was also invited to be presented at one of the top hacker conventions – ShmooCon 2024. arXiv