1 Repo
Awesome GitHub RepositoriesEvaluation Task Executors
Executes user-defined evaluation tasks against a model via the CLI with a custom task file.
Distinct from Custom Task Executors: Distinct from Custom Task Executors: specifically executes evaluation tasks against language models, not general automation logic.
Explore 1 awesome GitHub repository matching development tools & productivity · Evaluation Task Executors. Refine with filters or upvote what's useful.
Awesome Evaluation Task Executors GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- huggingface/lighteval
huggingface/lighteval
2,453Auf GitHub ansehenLighteval is an open-source framework for running standardized benchmarks and custom evaluation tasks against language models. It provides a system for defining new evaluation tasks with custom prompts, metrics, and scoring in YAML configuration files, and integrates with the Hugging Face Hub for storing and comparing results. The framework supports evaluating models across multiple inference backends, including transformers, vllm, and custom APIs, through a unified generation and log-probability interface. It includes a pluggable metric registry for built-in and custom scoring, a prediction
Executes user-defined evaluation tasks against a model using the CLI with a custom task file.
Pythonevaluationevaluation-frameworkevaluation-metrics
