Open-source alternatives to BEAF

This repository is no longer maintained and deprecated in favour of the repository at https://github.com/dsbowen/strong_reject. Please refer to that repository for full paper replication including human evaluation details and using the fine-tuned version of the StrongReject evaluator.

Authors: Liwei Jiang, Kavel Rao ⭐, Seungju Han ⭐, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, Nouha Dziri ⭐ Co-second authors

✨✨CVPR 2025 Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

.svg" alt="Documentation Status">

Train-val-test splits of GPT2Shape dataset can be found in folder gpt2shape: train val test

This repo provides the source code & data of our paper: Evaluating Object Hallucination in Large Vision-Language Models (EMNLP 2023).

This is the official repository of our paper ScanQA: 3D Question Answering for Spatial Scene Understanding (CVPR 2022) by Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoki Kawanabe. We propose a new 3D spatial understanding task for 3D question answering (3D-QA). In the 3D-QA task, models…

🤗 Dataset | 📖 arXiv | GitHub Atsuyuki Miyai 1   Jingkang Yang 2   Jingyang Zhang 3   Yifei Ming 4   Qing Yu 1,5   Go Irie 6   Sharon Yixuan Li 4   Hai Li 3   Ziwei Liu 2 Kiyoharu Aizawa 1 1 The University of Tokyo  2 S-Lab, Nanyang Technological…

FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models

English | 简体中文

Welcome to JailbreakZoo, a dedicated repository focused on the jailbreaking of large models (LMs), encompassing both large language models (LLMs) and vision language models (VLMs). This project aims to explore the vulnerabilities, exploit methods, and defense mechanisms associated with these…

📰 Latest News 📰 - 🗡️ What is HarmBench 🛡️ - 🌐 Overview 🌐 - ☕ Quick Start ☕ - ⚙️ Installation - 🛠️ Running the Evaluation Pipeline - ➕ Using your own models in HarmBench - ➕ Using your own red teaming methods in HarmBench - 🤗 Classifiers - ⚓ Documentation ⚓ - 🌱 HarmBench's Roadmap 🌱 -…

ICLR 2025 ChartMimic: Evaluating LMM’s Cross-Modal Reasoning Capability via Chart-to-Code Generation

NAACL 2025 Source code for MMEvalPro, a more trustworthy and efficient benchmark for evaluating LMMs

The jailbreak-evaluation is an easy-to-use Python package for language model jailbreak evaluation. The jailbreak-evaluation is designed for comprehensive and accurate evaluation of language model jailbreak attempts. Currently, jailbreak-evaluation support evaluating a language model jailbreak…

🍎 Project Page 📖 arXiv Paper 📊 Dataset🏆 Leaderboard

Data and code for paper "M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models"

ECCV 2024 BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models

We introduce the task of dense captioning in 3D scans from commodity RGB-D sensors. As input, we assume a point cloud of a 3D scene; the expected output is the bounding boxes along with the descriptions for the underlying objects. To address the 3D object detection and description problems, we…

Paper | Github | Dataset | Model

An easy-to-use Python framework to generate adversarial jailbreak prompts.

This repository contains the code for the paper "Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks" by Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury, accepted at LREC-CoLING 2024