Benchmark-uri pentru evaluarea SLM
For a framework for evaluating small language models, the strongest matches are internlm/opencompass (OpenCompass is a general LLM evaluation platform that supports), openai/evals (OpenAI Evals is a general-purpose framework for automating and) and eleutherai/lm-evaluation-harness (This framework provides standardized benchmarks and evaluation pipelines for). open-compass/opencompass and openai/simple-evals round out the shortlist. Each is ranked by relevance to your query, popularity and recent activity.
Compară cele mai bune benchmark-uri pentru evaluarea SLM. Am clasat aceste instrumente open-source după activitate și acuratețe pentru a te ajuta să găsești soluția potrivită pentru modelul tău.
Benchmark-uri pentru evaluarea SLM
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- internlm/opencompass
InternLM/opencompass
7,096Vezi pe GitHubOpenCompass is a comprehensive evaluation platform, benchmarking suite, and distributed model evaluator designed to measure the performance and accuracy of large language models. It provides a framework for benchmarking both open-source and API-based models against diverse datasets using standardized metrics and reproducible pipelines. The project features an automated judging framework that uses language models as judges to score and verify the quality of generated text. It includes a performance leaderboard system for comparing the relative capabilities of various models across industry-sta
OpenCompass is a general LLM evaluation platform that supports standardized benchmarks, multiple backends like Hugging Face, and performance metrics such as accuracy and perplexity, making it well-suited for evaluating small language models despite its primary focus on larger ones.
PythonModel Benchmarking SuitesModel Evaluation Frameworks - openai/evals
openai/evals
18,702Vezi pe GitHubEvals is a framework designed for automating, managing, and executing repeatable benchmarking suites to analyze the quality and performance of language models. It provides a platform for running standardized tests to measure model accuracy and track behavioral changes over time. The system distinguishes itself through a modular architecture that uses a standardized adapter layer to normalize inputs and outputs, allowing different models to be swapped and tested interchangeably. It supports the creation of custom benchmarks using proprietary data, enabling quality assurance on sensitive tasks
OpenAI Evals is a general-purpose framework for automating and benchmarking language models with standardized tests and model-agnostic adapters, fitting the evaluation-tool intent but lacking specific focus on small models (<7B) and efficiency metrics like speed or FLOPs that the visitor likely needs.
PythonModel Benchmarking SuitesModel Evaluation Frameworks - eleutherai/lm-evaluation-harness
EleutherAI/lm-evaluation-harness
11,460Vezi pe GitHubThis project is a standardized framework for benchmarking large language models across a wide range of academic and reasoning datasets. It provides a platform for executing automated evaluation tasks to measure model accuracy and performance, ensuring consistent assessment through a structured configuration schema. The framework distinguishes itself by incorporating a dedicated utility for data decontamination, which identifies and removes overlapping training samples from evaluation sets to prevent data leakage. It also features a flexible task builder that allows users to define custom benc
This framework provides standardized benchmarks and evaluation pipelines for language models, making it suitable for assessing small models, though it does not specifically emphasize efficiency metrics or small-model optimizations.
PythonModel Evaluation Frameworks - open-compass/opencompass
open-compass/opencompass
6,678Vezi pe GitHubOpenCompass is an open-source framework for standardized benchmarking of large language models. It provides a configurable evaluation pipeline that supports both objective and subjective assessment, using a dual-engine architecture to handle closed-form answer comparison and open-ended response rating. The framework is designed as a modular platform where datasets, models, and metrics are composed through declarative YAML configuration files. The framework distinguishes itself through its extensible model integration layer, which supports custom models, HuggingFace models, and third-party API
OpenCompass is a general-purpose LLM evaluation framework supporting HuggingFace models and standard benchmarks like MMLU, so it can readily evaluate small language models even without explicit small-model specialization.
PythonModel Benchmarking SuitesModel Evaluation Frameworks - openai/simple-evals
openai/simple-evals
4,354Vezi pe GitHubThis project is a language model evaluation framework and benchmarking tool designed to measure the accuracy and performance of models across diverse datasets. It provides a system for implementing model-based graders, running standardized tests for mathematical reasoning, coding, and factuality, and calculating quantified performance metrics such as precision, recall, F1 scores, and pass-at-k. The framework utilizes model-based grading and rubrics to validate response quality against expert-defined criteria. It includes a multi-model benchmarking loop and a model-agnostic API interface to co
openai/simple-evals is a general language model evaluation framework that supports multiple benchmarks and metrics, making it suitable for evaluating small models even though it does not explicitly focus on efficiency or small-scale architectures.
PythonModel Evaluation Frameworks - stanford-crfm/helm
stanford-crfm/helm
2,828Vezi pe GitHubHolistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.
HELM is a holistic evaluation framework from Stanford that covers standardized benchmarks and accuracy metrics for language models, including smaller ones, and supports multiple backends like Hugging Face, though its efficiency measurement focus is less pronounced than its accuracy and fairness features.
PythonEvaluation FrameworksModel Evaluation and Benchmarking - huggingface/evaluate
huggingface/evaluate
2,455Vezi pe GitHub🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
Hugging Face Evaluate is a general-purpose evaluation library that can be used to assess small language models with metrics like accuracy and perplexity, but it does not include standardized benchmark suites like GLUE or MMLU out of the box.
PythonEvaluation FrameworksModel EvaluationModel Evaluation and Benchmarking
