# llm-attacks/llm-attacks

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/llm-attacks-llm-attacks).**

4,509 stars · 602 forks · Python · mit

## Links

- GitHub: https://github.com/llm-attacks/llm-attacks
- Homepage: https://llm-attacks.org/
- awesome-repositories: https://awesome-repositories.com/repository/llm-attacks-llm-attacks.md

## Description

This repository provides tools and methodologies for studying adversarial attacks on large language models. It focuses on understanding how carefully crafted inputs can manipulate or bypass the safety mechanisms of LLMs, enabling researchers to probe model vulnerabilities and improve their robustness. The project covers techniques for generating adversarial prompts, evaluating model responses under attack conditions, and analyzing the effectiveness of different attack strategies.

## Tags

### Security & Cryptography

- [Adversarial Input Generation](https://awesome-repositories.com/f/security-cryptography/adversarial-robustness-testing/adversarial-input-generation.md) — Generates gradient-based adversarial inputs to stress-test AI model safety alignments.
- [Adversarial Suffix Generation](https://awesome-repositories.com/f/security-cryptography/adversarial-robustness-testing/adversarial-input-generation/adversarial-suffix-generation.md) — Generates specific string sequences to append to queries that bypass safety filters. ([source](https://llm-attacks.org/))
- [Transferability Testing](https://awesome-repositories.com/f/security-cryptography/model-safety-filters/transferability-testing.md) — Validates if adversarial suffixes created on open-source models can bypass filters on closed-source models.

### Artificial Intelligence & ML

- [Model Experiment Execution](https://awesome-repositories.com/f/artificial-intelligence-ml/dataset-management/evaluation-datasets/model-experiment-execution.md) — Implements a system for running harmful prompts across multiple models to compare safety robustness.
- [LLM Evaluation Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/llm-evaluation-frameworks.md) — Provides a testing environment to quantify how often harmful prompts bypass safety filters.
- [Adversarial Robustness Testing](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-evaluation-and-validation/model-capability-assessment/adversarial-robustness-testing.md) — Quantifies the success rate of jailbreak attacks through batch experiments to evaluate model stability. ([source](https://cdn.jsdelivr.net/gh/llm-attacks/llm-attacks@main/README.md))
- [Model Benchmarking Suites](https://awesome-repositories.com/f/artificial-intelligence-ml/model-benchmarking-suites.md) — Runs batch experiments across multiple harmful behaviors to measure model vulnerability to attacks.
- [Model Red-Teaming](https://awesome-repositories.com/f/artificial-intelligence-ml/model-red-teaming.md) — Implements automated adversarial testing to detect safety failures and bypass safety filters in LLMs.
- [Adversarial Suffix Optimization](https://awesome-repositories.com/f/artificial-intelligence-ml/optimization-algorithms/gradient-descent-algorithms/adversarial-suffix-optimization.md) — Uses coordinate gradient descent to iteratively refine token sequences that maximize adversarial response probability.
- [Safety and Alignment Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/safety-and-alignment-frameworks.md) — Provides a framework for identifying alignment failures to ensure models do not generate harmful content.
- [Discrete Token Search](https://awesome-repositories.com/f/artificial-intelligence-ml/discrete-token-search.md) — Searches the model vocabulary for replacement tokens that align with calculated gradient vectors.
- [Gradient-Based Jailbreak Generators](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-based-jailbreak-generators.md) — Automatically optimizes string sequences using model gradients to trigger restricted responses from language models.

### Part of an Awesome List

- [Adversarial Attacks](https://awesome-repositories.com/f/awesome-lists/security/adversarial-attacks.md) — Provides a comprehensive toolkit for generating gradient-based adversarial prompts to evaluate model robustness.
- [Evaluation Benchmarks](https://awesome-repositories.com/f/awesome-lists/ai/evaluation-benchmarks.md) — Universal and transferable adversarial attacks on aligned models.
- [White Box Attacks](https://awesome-repositories.com/f/awesome-lists/ai/white-box-attacks.md) — Provides universal and transferable adversarial attacks on aligned models.