EleutherAI/lm-evaluation-harness
Lm Evaluation Harness
This project is a standardized framework for benchmarking large language models across a wide range of academic and reasoning datasets. It provides a platform for executing automated evaluation tasks to measure model accuracy and performance, ensuring consistent assessment through a structured configuration schema.
The framework distinguishes itself by incorporating a dedicated utility for data decontamination, which identifies and removes overlapping training samples from evaluation sets to prevent data leakage. It also features a flexible task builder that allows users to define custom benchmarks by specifying unique data sources, prompt structures, and modular scoring metrics.
The system supports large-scale testing by orchestrating distributed evaluation workloads across multiple compute nodes. It utilizes an abstracted interface to standardize communication with diverse model backends, facilitating systematic validation of model capabilities before deployment.
Features
- Large Language Models - Measures language model performance against standardized academic datasets to compare reasoning capabilities and accuracy.
- Model Benchmarking - Benchmarks large language models against standardized academic and reasoning datasets to compare performance across complex tasks.
- Model Benchmarking Frameworks - Executes automated evaluation tasks to compare the capabilities and accuracy of generative AI models.
- Model Evaluation Frameworks - Provides a standardized toolkit for measuring the performance of large language models across diverse academic and reasoning benchmarks.
- Data Decontamination Tools - Prevents data leakage by identifying and removing overlapping training samples from evaluation sets using string similarity scores.
- Model Abstractions - Provides a unified interface for interacting with diverse language model backends to standardize inference and logit extraction.
- Machine Learning Model APIs - Systematically tests and verifies model capabilities to ensure performance requirements are met before deployment.
- Agentic Frameworks and LLM Tools - Standardized framework for evaluating language model performance.
- Evaluation Frameworks - Framework for few-shot evaluation of language models.
- Model Evaluation - Framework for few-shot evaluation of language models.
- Model Evaluation and Benchmarking - Framework for testing generative models across various tasks.
- Custom Evaluation Judges - Enables the creation of custom benchmark tasks by specifying data sources, metrics, and prompt structures.
- Distributed Orchestration - Orchestrates distributed evaluation workloads across multiple compute nodes to parallelize large-scale benchmark execution.
- Scoring Pipelines - Computes evaluation results by passing model outputs through modular validation functions for accuracy and performance indicators.
- Prompt Templates - Transforms input data into model-ready prompts using a flexible engine that supports complex formatting and few-shot examples.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
