huggingface/lighteval
Lighteval
Lighteval is an open-source framework for running standardized benchmarks and custom evaluation tasks against language models. It provides a system for defining new evaluation tasks with custom prompts, metrics, and scoring in YAML configuration files, and integrates with the Hugging Face Hub for storing and comparing results.
The framework supports evaluating models across multiple inference backends, including transformers, vllm, and custom APIs, through a unified generation and log-probability interface. It includes a pluggable metric registry for built-in and custom scoring, a prediction cache layer that avoids re-computation on repeated runs, and the ability to push results to remote storage destinations like the Hugging Face Hub, S3, or local directories. Per-sample outputs are saved to enable detailed debugging and performance analysis after evaluation.
Evaluation tasks are configured in YAML files specifying prompt templates, reference answers, and scoring metrics, with support for subset-based task splitting that generates separate configurations for each dataset subset automatically. The framework includes thousands of pre-built benchmark tasks and allows users to define custom tasks, custom metrics, and custom model implementations.
Features
- LLM Evaluation Frameworks - Provides an open-source framework for running standardized benchmarks and custom evaluation tasks against language models.
- Language Model Benchmark Suites - Runs pre-built and custom evaluation tasks against language models to measure performance.
- Custom Evaluation Judges - Builds new evaluation tasks with custom metrics and scoring to test model behavior on user-defined criteria.
- Model Performance Evaluators - Measures model outputs against predefined metrics to quantify accuracy and quality across diverse tasks.
- Unified Generation Interfaces - Provides a unified generation and log-probability interface across transformers, vllm, and custom API backends.
- Standardized Benchmarks - Executes thousands of ready-made evaluation tasks to measure model performance on standard tests.
- Model Evaluation - Runs models through a curated set of tasks and computes standard metrics to measure performance.
- YAML-Based Task Definitions - Defines evaluation tasks in YAML files specifying prompt templates, reference answers, and scoring metrics.
- Evaluation Task Definitions - Defines new evaluation tasks with custom prompts, metrics, and scoring in YAML configuration files.
- YAML Benchmark Definitions - Creates new benchmarks by specifying prompt templates, reference answers, and scoring metrics in YAML files.
- Custom Model Evaluation Adapters - Runs benchmark tasks against user-defined models by implementing subclasses with generation and log-probability methods.
- Evaluation Prediction Caches - Caches computed model outputs to disk or memory, skipping re-computation on repeated evaluation runs.
- Metric Selection and Creation - Selects built-in metrics or defines custom ones to score model outputs during evaluation.
- Evaluation Prompt Functions - Converts dataset lines into structured documents with query, choices, and gold index for evaluation.
- Evaluation Task Executors - Executes user-defined evaluation tasks against a model using the CLI with a custom task file.
- Inference Backend Abstraction - Supports running evaluations across transformers, vllm, and custom API backends through a unified interface.
- Inference Engine Evaluation Backends - Runs model evaluation using various inference engines like transformers, vllm, and custom APIs.
- Single-Subset Evaluation Configurations - Defines single-subset tasks with a prompt function, dataset source, splits, metrics, and generation parameters.
- Result Storage Integrations - Stores evaluation results on the Hugging Face Hub for sharing and comparing model performance.
- Custom Metric Registries - Ships a pluggable metric registry where built-in and custom scoring functions are registered by name.
- Datasets and Evaluation - Library for evaluating local LLMs on benchmarks.
- Evaluation Frameworks - Unified toolkit for benchmarking and evaluating language models.
- Model Evaluation - All-in-one toolkit for evaluating models across multiple backends.
- Model Evaluation and Benchmarking - Lightweight evaluation suite for language models.
