Awesome GitHub RepositoriesEvaluation Benchmarks

MMSegmentation is an open-source semantic segmentation toolbox built on PyTorch that provides a modular, configurable framework for building, training, evaluating, and deploying segmentation models. At its core, it offers a config-driven pipeline that assembles training, evaluation, and inference workflows by parsing hierarchical configuration files, with a modular component registry that enables plug-and-play composition of neural network modules, optimizers, datasets, and metrics. The framework supports the full model lifecycle through a unified runner interface that controls training, testi

Este proyecto es un conjunto estandarizado de problemas de abstracción y razonamiento diseñados para evaluar la capacidad de los modelos de inteligencia artificial para aprender nuevas reglas. Funciona como una prueba de inteligencia fluida y un benchmark de razonamiento, utilizando una colección de puzzles basados en rejillas y un dataset de síntesis de programas para evaluar cómo los agentes generan algoritmos a partir de ejemplos. El proyecto se centra en medir la inteligencia fluida general y la capacidad de generalización zero-shot, probando si un sistema puede aplicar la lógica aprendida a problemas no vistos sin depender de un entrenamiento específico para la tarea. Proporciona un framework para la investigación del razonamiento abstracto, evaluando específicamente la síntesis de programas y la capacidad de los modelos para descubrir patrones latentes para transformaciones de rejillas novedosas. El sistema incorpora una representación de dominio basada en rejillas y un espacio de búsqueda combinatorio de operaciones geométricas y topológicas. Incluye una interfaz human-in-the-loop que permite la construcción manual de rejillas de salida para definir la verdad fundamental para la validación y el benchmarking.

Youtu Agent is an open-source framework for building, running, and evaluating autonomous agents powered by large language models. It provides the core infrastructure for creating agents that follow reasoning loops, use toolkits, and coordinate with other agents to solve complex tasks, all managed through YAML-driven configuration files. The framework distinguishes itself through its support for multi-agent orchestration, where a planner agent decomposes tasks and coordinates specialized worker agents, and through its integration with the Model Context Protocol for connecting to external toolk

BioGPT is a biomedical large language model and domain-specific transformer designed for processing and creating specialized medical text. It functions as a generative tool and knowledge extraction engine trained on large-scale scientific literature to produce human-like scientific prose and factual responses to queries. The project provides specialized capabilities for biomedical named entity recognition and the extraction of complex relations from unstructured medical corpora. It is designed to identify and classify biological entities through data mining and relation extraction to support

This project is a language model evaluation framework and benchmarking tool designed to measure the accuracy and performance of models across diverse datasets. It provides a system for implementing model-based graders, running standardized tests for mathematical reasoning, coding, and factuality, and calculating quantified performance metrics such as precision, recall, F1 scores, and pass-at-k. The framework utilizes model-based grading and rubrics to validate response quality against expert-defined criteria. It includes a multi-model benchmarking loop and a model-agnostic API interface to co

AdalFlow es un framework de agentes de IA autónomos y una librería de aplicaciones LLM diseñada para construir flujos de trabajo modulares. Sirve como una interfaz agnóstica al modelo y orquestador de pipelines RAG, permitiendo a los usuarios desarrollar agentes ReAct que utilizan razonamiento iterativo y ejecución de herramientas externas para resolver tareas complejas. El proyecto se distingue por un sistema de optimización de prompts que utiliza descenso de gradiente textual para refinar automáticamente las plantillas de prompts y ejemplos de pocos disparos (few-shot). Trata la retroalimentación del modelo como una señal diferenciable, permitiendo una forma de retropropagación de LLM para mejorar iterativamente la calidad de la salida basada en métricas de evaluación. El framework cubre una amplia superficie de capacidades, incluyendo generación aumentada por recuperación (RAG) con búsqueda semántica vectorial y reranking, rastreo de ejecución basado en spans para observabilidad y análisis estructurado basado en esquemas. Proporciona una capa de comunicación unificada para numerosos proveedores de modelos propietarios y de código abierto, y admite la conversión de funciones de Python en interfaces de herramientas estandarizadas. El sistema está implementado en Python y se integra con MLflow para el seguimiento y análisis de flujos de trabajo.

Lighteval is an open-source framework for running standardized benchmarks and custom evaluation tasks against language models. It provides a system for defining new evaluation tasks with custom prompts, metrics, and scoring in YAML configuration files, and integrates with the Hugging Face Hub for storing and comparing results. The framework supports evaluating models across multiple inference backends, including transformers, vllm, and custom APIs, through a unified generation and log-probability interface. It includes a pluggable metric registry for built-in and custom scoring, a prediction

Official github repo for C-Eval, a Chinese evaluation suite for foundation models NeurIPS 2023

Natural Questions is a large-scale machine learning research dataset designed for training and evaluating open-domain question answering systems. It consists of a corpus of real search queries paired with human-annotated Wikipedia document spans, providing a standardized foundation for advancing automated information retrieval and comprehension technologies. The project distinguishes itself by providing high-quality ground truth data that supports multiple answer formats, including binary, short-form, and long-form responses. By incorporating extractive span annotations and structured documen

This repository introduces PIXIU, an open-source resource featuring the first financial large language models (LLMs), instruction tuning data, and evaluation benchmarks to holistically assess financial LLMs. Our goal is to continually push forward the open-source development of financial artificial intelligence (AI).

CBLUE1 中文医疗信息处理基准CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

CMMLU: Measuring massive multitask language understanding in Chinese

This repository contains information about AGIEval, data, code and output of baseline systems for the benchmark.

Industrial-first evaluation benchmark for LLMs in the DevOps/AIOps domain.

An open science effort to benchmark legal reasoning in foundation models

Benchmarking Legal Knowledge of Large Language Models

PromptCBLUE: a large-scale instruction-tuning dataset for multi-task and few-shot learning in the medical domain in Chinese

Abstract: FinanceBench is a first-of-its-kind test suite for evaluating the performance of LLMs on open book financial question answering (QA). This repository contains an open source sample of 150 annotated examples used in the evaluation and analysis of models assessed in the FinanceBench…

论文链接：https://arxiv.org/abs/2302.09432

The FinEval financial domain evaluation benchmark, based on quantitative fundamental methods and developed through long-term objective research, summarization, and rigorous manual screening, utilizes over 26,000 diverse question types that are highly consistent with real-world application scenarios.