InternLM/opencompass

Features

• Model Evaluation Frameworks - Provides a comprehensive framework for running model inference and validation on standardized datasets.
• LLM Evaluation - Provides a comprehensive platform for measuring the quality of LLM outputs using automated judges and custom metrics.
• Multi-Node Inference Scaling - Distributes evaluation tasks across multiple GPUs and nodes to handle workloads exceeding single-device memory.
• Inference Scaling - Distributes the computational workload of evaluating massive models across multiple GPUs and clusters to reduce processing time.
• LLM Benchmarking - Measures the accuracy and capabilities of large language models using standardized datasets and reproducible metrics.
• LLM Evaluation Frameworks - Offers a framework for benchmarking large language models against diverse datasets using standardized metrics and reproducible pipelines.
• Model Performance Benchmarking - Employs standardized tests to evaluate model speed and accuracy across diverse datasets.
• Model Benchmarking Suites - Ships a collection of tools to evaluate the accuracy, reasoning, and performance of LLMs against standardized datasets.
• Model Performance Leaderboards - Features a performance leaderboard system to compare the relative capabilities of open-source and proprietary models.
• Provider-Agnostic Model Interfaces - Provides a unified interface that wraps diverse model APIs and local weights for consistent input and output handling.
• Distributed Task Orchestration - Provides a system for defining and executing evaluation workloads across clusters of computing resources to reduce inference time.
• Prediction Workload Distribution - Splits massive evaluation workloads across computing clusters to process billion-scale models efficiently.
• LLM-As-A-Judge Scoring - Implements an automated judging framework where high-capability language models score generated responses based on predefined rubrics.
• Model Evaluation - Provides a scalable infrastructure for running massive model assessments across multiple GPUs and computing clusters.
• Model-Based Extraction - Employs secondary AI models to parse and isolate model outputs for a more accurate representation of capabilities.
• Answer Extraction Logics - Uses specialized models or regular expressions to isolate final answers from verbose model outputs for metric calculation.
• LLM Experiment Management - Records full experiment details via configuration files and reports resulting metrics in real time.
• Adversarial Robustness Testing - Tests model stability and security by applying various attack methods and evaluating tool-use capabilities.
• Evaluation Workflow Orchestrations - Sequences multiple evaluators in a custom workflow to assess complex scenarios through a multi-stage mechanism.
• Evaluation Configurations - Configures zero-shot, few-shot, and chain-of-thought prompting templates to standardize and maximize model performance during testing.
• Reasoning Verifications - Validates the logical steps and final answers of mathematical or complex reasoning tasks using specialized verification tools.
• Architecture Benchmarking - Evaluates model performance across a wide range of architectures to assess general capabilities.
• Dataset Preparation Tools - Automates the downloading and preparation of required evaluation datasets from remote storage servers or third-party hubs.
• Benchmark Dataset Loaders - Includes utilities for loading and preprocessing diverse benchmark datasets from remote hubs via a standardized interface.
• Evaluation Chains - Sequences prompting, generation, and verification steps into linear chains to assess complex reasoning tasks.
• Mathematical Verification - Validates the logic and accuracy of mathematical solutions through specialized verification steps.
• Configuration-Driven Pipelines - Defines evaluation workflows and dataset parameters through static files to ensure reproducibility across experiments.
• Vision-Language Model Benchmarking - Includes toolkits for the standardized evaluation of accuracy and reasoning in vision-language models.
• Unified Model Wrappers - Standardizes diverse open-source and API-based model interfaces under a single consistent configuration.
• Model Evaluation - One-stop evaluation platform supporting multiple datasets and distributed testing.

Star history