Awesome GitHub RepositoriesEvaluation Benchmarks

JARVIS is a system for large language model task orchestration, deployment management, and automation benchmarking. It utilizes a task orchestrator to decompose complex requests into actionable steps and coordinates various expert models to synthesize final responses. The project includes an AI model deployment manager to handle the local deployment of expert models across different hardware scales. It further provides an AI workflow API consisting of web endpoints used to trigger automated task workflows and retrieve results from model selection stages. The framework incorporates an automat

This project is a technical curriculum and development guide focused on large language model prompt engineering, fine-tuning, and the creation of retrieval augmented generation applications. It serves as a comprehensive resource for developers to master crafting precise instructions and textual patterns to improve the quality and predictability of model outputs. The material covers the end-to-end workflow of adapting open-source models to specific datasets and integrating language models with vector databases to generate responses based on private information. It also provides a systematic ap

This repository provides tools and methodologies for studying adversarial attacks on large language models. It focuses on understanding how carefully crafted inputs can manipulate or bypass the safety mechanisms of LLMs, enabling researchers to probe model vulnerabilities and improve their robustness. The project covers techniques for generating adversarial prompts, evaluating model responses under attack conditions, and analyzing the effectiveness of different attack strategies.

Malmo is a voxel-based simulation platform designed for artificial intelligence research and the study of autonomous agent behaviors. Built as a sandbox environment using Minecraft, it serves as a framework for multi-agent simulation and reinforcement learning research within a 3D grid of blocks. The project distinguishes itself through a multi-agent simulation framework that coordinates and synchronizes multiple autonomous agents to perform collaborative missions. It provides a standardized interface following reinforcement learning specifications, allowing it to function as an environment f

This project is a comprehensive ecosystem of frameworks, toolkits, and datasets designed to evaluate model vulnerabilities and analyze jailbreak patterns. It serves as an adversarial testing framework and research toolkit for measuring the effectiveness of safety guardrails in large language models. The system includes a library of real-world prompt injection datasets harvested from social media to study bypass strategies. It provides specialized tools for semantic attack analysis and prompt visualization, allowing for the mapping of relationships between adversarial prompts to discover commo

Acontext is an LLM orchestration backend and agent memory framework designed to manage session state and knowledge for AI agents. It functions as a context manager and orchestration layer that integrates model providers with a secure code sandbox and a zero-knowledge data store. The project is distinguished by its approach to knowledge distillation, capturing agent learnings as reusable Markdown skills and structured memory files. It provides a secure execution environment where shell commands and scripts run in isolated containers with the ability to mount these persistent skill files direct

EasyEdit is a framework and toolkit designed for updating, inserting, or erasing specific factual information within large language models without requiring full retraining. It functions as a parameter modifier and knowledge editing system capable of performing targeted weight updates across diverse model architectures. The project distinguishes itself by supporting both text-based and multimodal model editing, allowing for knowledge updates across image and text modalities. It provides utilities for model steering to adjust personality and reasoning patterns in real time via activation inter

OSWorld is an evaluation framework and multimodal agent benchmark designed to test the ability of large language models to complete complex tasks within virtualized operating system environments. It provides a virtualized desktop sandbox and a virtual machine orchestrator to deploy, snapshot, and reset cloud-based desktops, ensuring reproducible test states for AI agent interactions. The system distinguishes itself by providing an OS-level action space that translates model decisions into mouse clicks, keyboard inputs, and system commands. It employs a standardized interface to integrate vari

RLinf is a distributed reinforcement learning orchestrator and embodied AI training framework. It provides the infrastructure to train vision-language-action models and robotic policies using a combination of reinforcement learning and supervised fine-tuning. The system is designed for scaling workloads across GPU clusters, managing the placement of actors, rollout workers, and environment components. It features a specialized robotics data collection pipeline for gathering teleoperated demonstrations and simulation trajectories into standardized replay buffers, alongside a hardware interface

CVPR2024 Highlight VBench - We Evaluate Video Generation

This project provides a comprehensive guide and framework for implementing autonomous AI coding assistants within local development environments. It focuses on orchestrating multi-agent teams that can plan, execute, and verify complex software engineering tasks, such as refactoring, bug resolution, and test generation, while maintaining deep awareness of project-specific context and memory. The system distinguishes itself through a robust security-first architecture that enforces granular access controls, execution isolation, and mandatory human-in-the-loop approvals for all file modification

Chinese safety prompts for evaluating and improving the safety of LLMs. 中文安全prompts，用于评估和提升大模型的安全性。

📰 Latest News 📰 - 🗡️ What is HarmBench 🛡️ - 🌐 Overview 🌐 - ☕ Quick Start ☕ - ⚙️ Installation - 🛠️ Running the Evaluation Pipeline - ➕ Using your own models in HarmBench - ➕ Using your own red teaming methods in HarmBench - 🤗 Classifiers - ⚓ Documentation ⚓ - 🌱 HarmBench's Roadmap 🌱 -…

An easy-to-use Python framework to generate adversarial jailbreak prompts.

On the Hidden Mystery of OCR in Large Multimodal Models (OCRBench)

✨✨CVPR 2025 Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI

JailbreakBench -->