SWE-bench/SWE-bench
SWE Bench
SWE-bench is an automated evaluation framework that tests large language models on real-world software engineering tasks. It measures how effectively models can generate and apply code patches that resolve actual GitHub issues, using a standardized dataset and scoring system built around Docker-based patch verification against original project test suites.
The framework provides curated benchmark datasets spanning comprehensive, fast, verified, multilingual, and multimodal evaluation splits, allowing targeted assessment of model capabilities across different programming languages and issue types. It includes a containerized evaluation harness that can run locally or on cloud infrastructure, with support for BM25 retrieval indexing to identify relevant code context for bug-fixing tasks. The system parses test logs across multiple frameworks including Pytest, Jest, Maven, and Gradle to determine patch correctness, and generates unified diff patches for automated application to repository codebases.
Beyond evaluation, SWE-bench supports creating new benchmark tasks and training data from user-provided repositories, running live inference on individual GitHub issues through repository cloning and retrieval index construction, and comparing agent and model performance across variants using resolution rates, costs, and trajectories. The framework also provides tools for dataset tokenization, retrieval dataset loading, and text dataset generation for research contexts.
Features
- GitHub Issue Resolution Benchmarks - Measures how effectively models generate and apply code fixes to real software bugs using a standardized dataset and scoring system.
- Language Model Benchmark Suites - Tests language models on real-world software engineering tasks from GitHub issues.
- Code Patch Evaluations - Tests submitted code patches against real GitHub issue test suites.
- Docker-Verified Code Patch Benchmarks - Tests large language models on real-world GitHub issues using Docker-based patch verification against original project test suites.
- Ground-Truth Scoring - Compares generated patches against gold-standard patches and test cases to measure accuracy.
- Inference APIs - Runs model inference on benchmark instances via external APIs with progress tracking.
- Model Performance Benchmarking - Evaluates language models by measuring the percentage of real-world issues they resolve with patches.
- Cross-Language Code Benchmarks - Tests language models on software issues across multiple programming languages.
- Cross-Language Code Evaluations - Evaluates language models on software issues across multiple programming languages.
- Prebuilt Evaluation Containers - Uses a container-based harness to ensure consistent and repeatable patch verification across environments.
- Evaluation Harness Containers - Builds isolated Docker containers for each software issue to verify generated patches.
- Benchmark Evaluation Runners - Runs the evaluation harness on supported dataset splits to score model-generated patches against real issues.
- Docker-Based Patch Verification Harnesses - Provides a containerized system that builds task-specific environments to verify generated patches against original test suites.
- Evaluation Pipelines - Executes a standardized pipeline that builds Docker images, runs patch predictions, and logs results.
- Unified Diff Formats - Generates and refines code patches as unified diffs for automated application to codebases.
- LLM-As-A-Judge Scoring - Scores model-generated patches against real project test suites.
- Automated Bug Fixing Evaluation - Automatically generates and tests code patches for real-world software bugs.
- Test Log Parsers - Parses test logs from Pytest, Jest, Maven, and Gradle to determine patch correctness.
- Code Context Retrieval - Uses BM25 retrieval to index repository documents for accurate bug fixes.
- Local Harness Runners - Executes a command-line tool that builds Docker images, runs patch verification, and stores logs locally.
- Multi-Subset Evaluators - Runs models against curated benchmarks including human-verified, multilingual, lite, and multimodal issue sets.
- BM25 Search Indices - Performs BM25 retrieval on datasets to find relevant documents for a given query.
- Benchmark Dataset Loaders - Loads pre-built datasets of real-world software issues with variants for targeted evaluation.
- Multi-Split Dataset Loaders - Provides curated problem sets including comprehensive, fast, verified, and multimodal evaluation splits.
- Automated Issue Resolvers - Assesses model performance on software issues with visual elements.
- Multimodal Issue Resolvers - Evaluates models on software issues with visual elements like screenshots.
- Dataset Loaders - Loads pre-built benchmark datasets from Hugging Face for comprehensive and multimodal evaluation.
- Agent Run Comparators - Compares agent configurations and models side-by-side on standardized tasks.
- Benchmark Result Comparison - Compares resolution rates, costs, and trajectories across model variants.
- Multi-Framework - Parses test logs from frameworks including Pytest, Jest, Maven, and Gradle for cross-language evaluation.
- Coding Benchmarks - Benchmark for resolving real-world GitHub issues.
