Benchmarks für SLM-Evaluierung
For a framework for evaluating small language models, the strongest matches are internlm/opencompass (OpenCompass is a general LLM evaluation platform that supports), openai/evals (OpenAI Evals is a general-purpose framework for automating and) and eleutherai/lm-evaluation-harness (This framework provides standardized benchmarks and evaluation pipelines for). open-compass/opencompass and openai/simple-evals round out the shortlist. Each is ranked by relevance to your query, popularity and recent activity.
Vergleiche die besten Benchmarks für SLM-Evaluation. Wir haben diese Open-Source-Tools nach Aktivität und Genauigkeit bewertet, damit du die beste Lösung für dein Modell findest.
Benchmarks für SLM-Evaluierung
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- internlm/opencompass
InternLM/opencompass
7,096Auf GitHub ansehenOpenCompass is a comprehensive evaluation platform, benchmarking suite, and distributed model evaluator designed to measure the performance and accuracy of large language models. It provides a framework for benchmarking both open-source and API-based models against diverse datasets using standardized metrics and reproducible pipelines. The project features an automated judging framework that uses language models as judges to score and verify the quality of generated text. It includes a performance leaderboard system for comparing the relative capabilities of various models across industry-sta
OpenCompass is a general LLM evaluation platform that supports standardized benchmarks, multiple backends like Hugging Face, and performance metrics such as accuracy and perplexity, making it well-suited for evaluating small language models despite its primary focus on larger ones.
PythonModel Benchmarking SuitesModel Evaluation Frameworks - openai/evals
openai/evals
18,702Auf GitHub ansehenEvals is a framework designed for automating, managing, and executing repeatable benchmarking suites to analyze the quality and performance of language models. It provides a platform for running standardized tests to measure model accuracy and track behavioral changes over time. The system distinguishes itself through a modular architecture that uses a standardized adapter layer to normalize inputs and outputs, allowing different models to be swapped and tested interchangeably. It supports the creation of custom benchmarks using proprietary data, enabling quality assurance on sensitive tasks
OpenAI Evals is a general-purpose framework for automating and benchmarking language models with standardized tests and model-agnostic adapters, fitting the evaluation-tool intent but lacking specific focus on small models (<7B) and efficiency metrics like speed or FLOPs that the visitor likely needs.
PythonModel Benchmarking SuitesModel Evaluation Frameworks - eleutherai/lm-evaluation-harness
EleutherAI/lm-evaluation-harness
11,460Auf GitHub ansehenThis project is a standardized framework for benchmarking large language models across a wide range of academic and reasoning datasets. It provides a platform for executing automated evaluation tasks to measure model accuracy and performance, ensuring consistent assessment through a structured configuration schema. The framework distinguishes itself by incorporating a dedicated utility for data decontamination, which identifies and removes overlapping training samples from evaluation sets to prevent data leakage. It also features a flexible task builder that allows users to define custom benc
This framework provides standardized benchmarks and evaluation pipelines for language models, making it suitable for assessing small models, though it does not specifically emphasize efficiency metrics or small-model optimizations.
PythonModel Evaluation Frameworks - open-compass/opencompass
open-compass/opencompass
6,678Auf GitHub ansehenOpenCompass is an open-source framework for standardized benchmarking of large language models. It provides a configurable evaluation pipeline that supports both objective and subjective assessment, using a dual-engine architecture to handle closed-form answer comparison and open-ended response rating. The framework is designed as a modular platform where datasets, models, and metrics are composed through declarative YAML configuration files. The framework distinguishes itself through its extensible model integration layer, which supports custom models, HuggingFace models, and third-party API
OpenCompass is a general-purpose LLM evaluation framework supporting HuggingFace models and standard benchmarks like MMLU, so it can readily evaluate small language models even without explicit small-model specialization.
PythonModel Benchmarking SuitesModel Evaluation Frameworks - openai/simple-evals
openai/simple-evals
4,354Auf GitHub ansehenThis project is a language model evaluation framework and benchmarking tool designed to measure the accuracy and performance of models across diverse datasets. It provides a system for implementing model-based graders, running standardized tests for mathematical reasoning, coding, and factuality, and calculating quantified performance metrics such as precision, recall, F1 scores, and pass-at-k. The framework utilizes model-based grading and rubrics to validate response quality against expert-defined criteria. It includes a multi-model benchmarking loop and a model-agnostic API interface to co
openai/simple-evals is a general language model evaluation framework that supports multiple benchmarks and metrics, making it suitable for evaluating small models even though it does not explicitly focus on efficiency or small-scale architectures.
PythonModel Evaluation Frameworks - stanford-crfm/helm
stanford-crfm/helm
2,828Auf GitHub ansehenHolistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.
HELM is a holistic evaluation framework from Stanford that covers standardized benchmarks and accuracy metrics for language models, including smaller ones, and supports multiple backends like Hugging Face, though its efficiency measurement focus is less pronounced than its accuracy and fairness features.
PythonEvaluation FrameworksModel Evaluation and Benchmarking - huggingface/evaluate
huggingface/evaluate
2,455Auf GitHub ansehen🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
Hugging Face Evaluate is a general-purpose evaluation library that can be used to assess small language models with metrics like accuracy and perplexity, but it does not include standardized benchmark suites like GLUE or MMLU out of the box.
PythonEvaluation FrameworksModel EvaluationModel Evaluation and Benchmarking
