Awesome GitHub RepositoriesBenchmark Data Generators

Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing

The 1BRC (One Billion Row Challenge) is a Java performance benchmarking exercise that processes one billion temperature records from a text file to compute the minimum, mean, and maximum temperature per weather station. At its core, it is a large-scale data aggregation challenge designed to test how efficiently a Java program can parse and aggregate structured data from a plain text file, serving as both a programming exercise and a benchmark for Java performance optimization. The project distinguishes itself through a collection of performance-oriented architectural patterns for high-through

Vespa is a distributed search engine, vector database, and machine learning ranking engine. It serves as an AI search platform designed to handle large-scale document indexing and complex query processing across a cluster of nodes, combining keyword retrieval with high-dimensional embedding storage for semantic similarity search. The platform distinguishes itself by integrating machine learning models directly into the search pipeline to perform real-time inference and ranking. It converts these models into ranking expressions to score and order results based on relevance, while providing a s

Criterion es una biblioteca de microbenchmarking basada en estadísticas y una herramienta de regresión de rendimiento para Rust. Proporciona un framework para aislar y medir pequeños segmentos de código, utilizando análisis estadístico para eliminar el ruido y garantizar mediciones fiables y repetibles de la velocidad de ejecución. La herramienta destaca por su suite de visualización de rendimiento, que genera informes HTML y gráficos para seguir las tendencias de rendimiento y el throughput. Incluye un sistema para comparar los tiempos de ejecución actuales con líneas base almacenadas para identificar y prevenir caídas de rendimiento. La biblioteca cubre la medición de funciones asíncronas, benchmarking parametrizado para escalado de entrada y cálculo de throughput de código. También admite la integración de métricas de hardware personalizadas y contadores de procesador para capturar datos de bajo nivel durante las ejecuciones. La automatización es compatible mediante una interfaz de línea de comandos para el filtrado de benchmarks y un modo de validación para verificar la ejecución exitosa dentro de pipelines de integración continua.

SWE-bench is an automated evaluation framework that tests large language models on real-world software engineering tasks. It measures how effectively models can generate and apply code patches that resolve actual GitHub issues, using a standardized dataset and scoring system built around Docker-based patch verification against original project test suites. The framework provides curated benchmark datasets spanning comprehensive, fast, verified, multilingual, and multimodal evaluation splits, allowing targeted assessment of model capabilities across different programming languages and issue ty

TransformerLens is a library for mechanistic interpretability research designed to reverse engineer the learned algorithms within large language models. It provides a standardized framework for wrapping diverse transformer architectures, allowing researchers to extract, manipulate, and analyze internal activations and weights through a consistent interface. The project distinguishes itself through a comprehensive system of activation hooks that can capture, patch, and ablate internal tensors during the forward pass. It includes specialized utilities for decomposing fused projections, material