Awesome GitHub RepositoriesBenchmark Data Generators

Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing

The 1BRC (One Billion Row Challenge) is a Java performance benchmarking exercise that processes one billion temperature records from a text file to compute the minimum, mean, and maximum temperature per weather station. At its core, it is a large-scale data aggregation challenge designed to test how efficiently a Java program can parse and aggregate structured data from a plain text file, serving as both a programming exercise and a benchmark for Java performance optimization. The project distinguishes itself through a collection of performance-oriented architectural patterns for high-through

Vespa is a distributed search engine, vector database, and machine learning ranking engine. It serves as an AI search platform designed to handle large-scale document indexing and complex query processing across a cluster of nodes, combining keyword retrieval with high-dimensional embedding storage for semantic similarity search. The platform distinguishes itself by integrating machine learning models directly into the search pipeline to perform real-time inference and ranking. It converts these models into ranking expressions to score and order results based on relevance, while providing a s

Criterion est une bibliothèque de micro-benchmarking axée sur les statistiques et un outil de régression de performance pour Rust. Elle fournit un framework pour isoler et mesurer de petits segments de code, en utilisant l'analyse statistique pour éliminer le bruit et garantir des mesures de vitesse d'exécution fiables et reproductibles. L'outil se distingue par une suite de visualisation des performances qui génère des rapports HTML et des graphiques pour suivre les tendances de performance et le débit. Il inclut un système pour comparer les temps d'exécution actuels aux lignes de base enregistrées afin d'identifier et de prévenir les baisses de performance. La bibliothèque couvre la mesure de fonctions asynchrones, le benchmarking paramétré pour la mise à l'échelle des entrées et le calcul du débit de code. Elle prend également en charge l'intégration de métriques matérielles personnalisées et de compteurs de processeur pour capturer des données de bas niveau pendant les exécutions. L'automatisation est prise en charge via une interface en ligne de commande pour le filtrage des benchmarks et un mode de validation pour vérifier l'exécution réussie au sein des pipelines d'intégration continue.

SWE-bench is an automated evaluation framework that tests large language models on real-world software engineering tasks. It measures how effectively models can generate and apply code patches that resolve actual GitHub issues, using a standardized dataset and scoring system built around Docker-based patch verification against original project test suites. The framework provides curated benchmark datasets spanning comprehensive, fast, verified, multilingual, and multimodal evaluation splits, allowing targeted assessment of model capabilities across different programming languages and issue ty

TransformerLens is a library for mechanistic interpretability research designed to reverse engineer the learned algorithms within large language models. It provides a standardized framework for wrapping diverse transformer architectures, allowing researchers to extract, manipulate, and analyze internal activations and weights through a consistent interface. The project distinguishes itself through a comprehensive system of activation hooks that can capture, patch, and ablate internal tensors during the forward pass. It includes specialized utilities for decomposing fused projections, material