EvolvingLMMs-Lab/lmms-eval

Features

• LLM Evaluation Frameworks - Provides a comprehensive benchmarking system for measuring large multimodal models across text, image, audio, and video.
• Model Performance Benchmarking - Provides a comprehensive system to evaluate the speed and accuracy of multimodal models across diverse datasets.
• Evaluation Dataset Structurers - Specifies datasets, input processing functions, and output types via configuration files to create structured benchmarks.
• Confidence Interval Calculators - Calculates confidence intervals, clustered standard errors, and p-values to ensure benchmark scores are reproducible.
• Model Backend Adapters - Wraps diverse model backends in a unified interface to ensure consistent data formats across benchmarks.
• Model Benchmarking Tools - Calculates accuracy and efficiency metrics for models processing combined visual, auditory, and textual inputs.
• Model Evaluation Suites - Ships a suite for computing statistical significance, throughput, and token usage for multimodal evaluations.
• Model Integration Interfaces - Wraps diverse model backends in a standard interface to enable consistent benchmarking across multimodal datasets.
• Model Performance Analysis - Quantifies model quality and efficiency by calculating accuracy, throughput, and statistical significance.
• Model Red-Teaming - Implements adversarial testing using red-teaming datasets to detect hallucinations, biases, and jailbreak vulnerabilities.
• Multimodal Message Containers - Structures text, image, video, and audio inputs into a standardized multimodal message protocol for consistent model consumption.
• Model Quality Metrics - Calculates accuracy, perplexity, and F1 scores using configurable aggregation methods to quantify model quality.
• Multimodal Input Tuples - Structures multimodal data into specific tuples to maintain a consistent contract between dataset components and models.
• Text Capability Benchmarks - Runs standard text-only language benchmarks to isolate linguistic reasoning from multimodal capabilities.
• Visual Mathematical Reasoning Evaluation - Evaluates a model's capacity to solve mathematical problems presented in visual formats.
• Visual Question Answering Evaluation - Evaluates performance on visual question answering, captioning, and comprehension across images.
• Visual Spatial Reasoning Evaluation - Benchmarks a model's understanding of object locations and physical spatial relations within scenes.
• Task Instance Definitions - Uses external manifests and registration files to define datasets, processing logic, and scoring metrics for benchmark tasks.
• Multimodal Dataset Loaders - Manages task configurations and processes raw multimodal samples into model-ready formats.
• Benchmark Task Management - Indexes available evaluation tasks, loads custom configurations, and automates dataset downloads.
• Semantic Video Understanding Tools - Measures temporal reasoning, action recognition, and long-form comprehension of video content.
• Token Consumption Trackers - Normalizes diverse model output formats into consistent token counts for input, output, and reasoning.
• Multimodal Reasoning Evaluations - Tests model reasoning capabilities over documents, spatial relationships, and complex mathematical problems using media inputs.
• Token Throughput Measurement - Calculates latency, token generation speed, and time to first token to evaluate inference performance.
• Multimodal Input Processors - Transforms raw dataset samples into visual, audio, or text formats required for model inference.
• Agentic Execution Loops - Orchestrates multi-round evaluations by iteratively sending prompts and processing outputs until a terminal signal is reached.
• Reasoning Block Filters - Removes internal reasoning blocks from model outputs before scoring to ensure metrics reflect the final answer.
• Custom Performance Metrics - Allows registration of custom scoring and aggregation functions to evaluate model performance via mathematical rules.
• Answer Accuracy Evaluators - Computes log-probabilities of target continuations to evaluate model accuracy on closed-set multiple-choice tasks.
• High Throughput Inference - Increases request speed using adaptive concurrency, prefix-aware queueing, and shared cache reuse for media.
• Inference Optimization Tools - Optimizes evaluation speed through request batching, response caching, and accelerated media decoding.
• Audio Performance Benchmarks - Assesses model capabilities in speech recognition, speech translation, and audio-based question answering.
• Remote Evaluation Execution - Hosts an HTTP server to asynchronously trigger and track long-running model evaluations via remote client requests.
• Programmatic Evaluation APIs - Provides a functional API to programmatically control model arguments, task selection, and batch sizes for automated benchmarking.
• Prompt-Based Text Generation - Generates text responses by combining prompts with media files to test natural language generation capabilities.
• Visual Document Understanding - Tests the ability to extract and reason over information from documents, infographics, and images with text.
• Evaluation Result Caches - Stores shared results in a directory or database to avoid redundant computations and reduce costs.
• Response Caching - Stores model outputs and evaluation results in a persistent database to prevent redundant API calls and costs.
• Parallel Media Decoding - Accelerates video and audio processing through parallel decoding and blob storage to reduce evaluation latency.
• Custom Metric Registries - Allows registration of custom mathematical rules and scoring functions to calculate performance metrics from model outputs.
• Batch Processing Monitors - Aggregates processing time and average throughput across concurrent requests to assess efficiency under load.
• Model Observability Suites - Logs all generated responses to a real-time file to provide observability into model behavior.
• Inference Efficiency Metrics - Aggregates sample-level token counts and scores into high-level summaries like tokens per correct answer.
• Model Evaluation and Benchmarking - Evaluation framework for large vision-language models.

Star history