Projects sharing features with GPTFuzz

In this paper, we conduct the first work to propose the concept of indirect jailbreak and achieve Retrieval-Augmented Generation (RAG) via LangChain. Building on this, we further design a novel method of indirect jailbreak attack, termed Poisoned-LangChain (PLC), which leverages a poisoned…

Chawin Sitawarin 1 Norman Mu 1 David Wagner 1 Alexandre Araujo 2

The respositiy is public package of the paper titled "Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues" submitted to ACL 2024.

📄 Paper • 🤗 Dataset

Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models

This repository is for our new work: "BadReasoner: Planting Tunable Overthinking Backdoors into Large Reasoning Models for Fun or Profit", feel free to propose your issues!! 😎

This is the source code for the ICLR 2024 paper "An LLM can Fool Itself: A Prompt-Based Adversarial Attack", Xilie Xu (NUS), Keyi Kong (SDU), Ning Liu (SDU), Lizhen Cui (SDU), Di Wang (KAUST), Jingfeng Zhang (University of Auckland/RIKEN-AIP), Mohan Kankanhalli (NUS).…

We present a new class of vulnerabilities and impacts stemming from "indirect prompt injection" affecting language models integrated with applications. Our demos currently span GPT-4 (Bing and synthetic apps) using ChatML, GPT-3 & LangChain based apps in addition to proof-of-concepts for attacks…

Paper Link: https://arxiv.org/pdf/2304.05197.pdf

This repository contains the code implementation for the paper CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models.

Logo generated by GPT-4

This is the replication package for the paper MASTERKEY: Automated Jailbreaking of Large Language Model Chatbots.

Code for our NAACL 2024 Paper "Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking"

PurpleLlama is a collection of security components and toolkits designed for large language models. It provides specialized systems including a code security scanner, a content moderation system, a prompt injection firewall, and a security assessment toolkit. The project enables the identification and blocking of jailbreaking attempts and malicious prompts during model inference. It includes capabilities for detecting violating content across multiple languages and modalities and scanning generated code for vulnerabilities to prevent the execution of insecure commands. The framework further

Official implementation for "Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy"

The official implementation of our NAACL 2024 paper "A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily".

Garak is an AI model evaluation tool and vulnerability scanner designed for red teaming large language models and auditing the security of retrieval-augmented generation pipelines. It identifies behavioral weaknesses, such as jailbreaks, hallucinations, and data leakage, by simulating adversarial attacks and executing automated testing vectors. The framework utilizes an adaptive probing loop where prompts can react to previous model behavior and be modified in flight via middleware. To ensure consistent analysis, it employs a provider-agnostic interface to interact with various model APIs and

NeMo-Guardrails is a toolkit for adding programmable safety constraints and dialogue boundaries to large language model conversational systems. It functions as security middleware that intercepts inputs and outputs to block prompt injections, jailbreaks, and sensitive data leaks, while providing a conversational dialogue manager to define structured interaction flows through configuration files. The framework includes a hallucination filter to screen model outputs for factual accuracy and a specialized modeling language for defining conversational flows and constraints. It provides capabiliti

LLM Guard is a security firewall and guardrail framework designed to scan and sanitize inputs and outputs for large language models. It functions as a proxy gateway and security layer to block prompt injections, toxicity, and sensitive data leakage while ensuring that model interactions remain compliant with organizational policies. The system distinguishes itself through a modular scanner pipeline that utilizes local model orchestration to eliminate external network dependencies. It supports real-time security filtering via streaming chunk analysis and implements a fail-fast execution model

LLM Prompt Injection Detector

This repository contains a PyTorch implementation for our paper Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation.

QROA, or Query-Response Optimization Attack, is an innovative and robust strategy designed to explore and exploit vulnerabilities in Large Language Models (LLMs) through black-box interactions. This method leverages optimized triggers embedded within benign-looking instructions to manipulate…

This repository contains code and data for "FUZZLLM: A Novel and Universal Fuzzing Framework for Discovering Jailbreak Vulnerabilities in LLMs" (accepted to ICASSP 2024). Our work was also invited to be presented at one of the top hacker conventions – ShmooCon 2024. arXiv

Abstract. While Large Language Models (LLMs) display versatile functionality, they continue to generate harmful, biased, and toxic content, as demonstrated by the prevalence of human-designed jailbreaks. In this work, we present Tree of Attacks with Pruning (TAP), an automated method for…