Open-source alternatives to Ann Benchmarks

Natural Questions is a large-scale machine learning research dataset designed for training and evaluating open-domain question answering systems. It consists of a corpus of real search queries paired with human-annotated Wikipedia document spans, providing a standardized foundation for advancing automated information retrieval and comprehension technologies. The project distinguishes itself by providing high-quality ground truth data that supports multiple answer formats, including binary, short-form, and long-form responses. By incorporating extractive span annotations and structured documen

This project is a Python wrapper for the TA-Lib library, providing a technical analysis library for computing moving averages, momentum, and volatility metrics for financial time series analysis. It serves as a financial indicator calculator that processes price and volume arrays to generate technical signals and pattern recognition. The library includes an incremental data processor capable of computing the most recent technical indicator values as new streaming market data arrives. This allows for real-time price monitoring and the processing of streaming data without recalculating entire d

Wandb is a centralized platform for machine learning experiment tracking, model registry management, and workflow orchestration. It provides a comprehensive suite of tools for logging, visualizing, and versioning training metrics, model artifacts, and hyperparameter sweeps to ensure reproducibility across development cycles. The platform also functions as an observability tool for large language model applications, enabling the tracing of execution steps, token usage, and reasoning processes. The project distinguishes itself through its event-driven automation capabilities, which allow users

dirsearch is a command-line security tool and web path scanner used for discovering hidden directories and files on web servers. It functions as a recursive directory fuzzer and brute-force utility that identifies undocumented paths and sensitive files using wordlists and HTTP status codes. The tool distinguishes itself through template-driven path generation and an automated HTTP response filter that uses status codes, content length, and regex patterns to isolate valid targets. It supports recursive directory crawling to map complex web structures and provides state-persistence serializatio

Pygame is a framework for building interactive 2D applications and games using the Python programming language. It functions as a 2D graphics rendering engine, a game input and event manager, and a multimedia audio toolkit. The project serves as a wrapper for the Simple DirectMedia Layer, providing a Python interface for low-level access to graphics, audio, keyboard, mouse, and joystick hardware. Its capabilities cover 2D graphics rendering and image manipulation, digital audio playback and streaming, and the management of game entities and physics simulations. It also includes tools for rea

doggo is a multi-protocol DNS lookup client, performance analyzer, and diagnostic troubleshooting tool. It functions as a command-line utility for performing DNS queries across various transport protocols, including UDP, TCP, HTTPS, TLS, and QUIC. The project distinguishes itself through a focus on secure and distributed resolution. It implements encrypted DNS standards such as DNS over HTTPS, DNS over QUIC, and DNSCrypt, and provides DNSSEC validation to verify chains of trust for signed domains. Additionally, it can execute queries from global geographic locations via an external API to ana

ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself

Hydra is a hierarchical configuration framework and type-safe configuration manager. It is designed to manage complex application settings through composable configuration files and command-line overrides, ensuring that configuration values match expected data types during instantiation. The framework functions as a dynamic object instantiator that creates class instances directly from hierarchical configuration values and nested objects. It also operates as a hyperparameter sweep orchestrator and cluster job launcher, enabling the execution of multiple application runs across parameter range

A lightweight client for managing MariaDB, MySQL, SQL Server, PostgreSQL, SQLite, Interbase and Firebird, written in Delphi and Lazarus/FreePascal

Storm is an SSH connection manager and configuration tool that serves as a cross-platform wrapper for SSH clients. It provides a system for organizing, editing, and searching remote server host configurations and custom SSH directives. The project implements a multi-interface bridge that allows users to manage connection settings through a command line interface, web environments, and desktop interfaces. It uses alias-based command mapping to link short-hand identifiers to complex connection strings. The tool includes a Python library that allows for the programmatic automation of SSH connec

TransformerLab is an MLOps orchestration platform and research environment designed for the training, fine-tuning, and evaluation of large language models. It serves as a centralized control plane for managing machine learning jobs and coordinating distributed GPU compute across hybrid cloud and on-premise providers. The platform distinguishes itself through agent-driven model optimization, using AI assistants to analyze metrics and automatically propose and queue hyperparameter experiments. It provides a remote development environment that allows users to launch interactive notebooks, code e

This project is a language model evaluation framework and benchmarking tool designed to measure the accuracy and performance of models across diverse datasets. It provides a system for implementing model-based graders, running standardized tests for mathematical reasoning, coding, and factuality, and calculating quantified performance metrics such as precision, recall, F1 scores, and pass-at-k. The framework utilizes model-based grading and rubrics to validate response quality against expert-defined criteria. It includes a multi-model benchmarking loop and a model-agnostic API interface to co

MMSegmentation is an open-source semantic segmentation toolbox built on PyTorch that provides a modular, configurable framework for building, training, evaluating, and deploying segmentation models. At its core, it offers a config-driven pipeline that assembles training, evaluation, and inference workflows by parsing hierarchical configuration files, with a modular component registry that enables plug-and-play composition of neural network modules, optimizers, datasets, and metrics. The framework supports the full model lifecycle through a unified runner interface that controls training, testi

ParlAI is a conversational AI research framework designed for training, evaluating, and sharing dialogue models using a unified interface for datasets and agents. It functions as a PyTorch-based training platform and a dialogue data collection system, providing a centralized model zoo for the distribution of versioned pretrained agents. The project distinguishes itself through a knowledge-grounded retrieval system that combines dense and sparse indexing to ground responses in external information. It also provides a comprehensive infrastructure for gathering human-AI interaction data via inte

This project is an educational resource and engineering guide for building, deploying, and optimizing large language model applications and production pipelines. It serves as a blueprint for cloud AI infrastructure, providing a framework for orchestrating inference endpoints, data warehouses, and scalable production environments. The repository provides specific implementation patterns for retrieval augmented generation to ground model responses in external data. It includes a training workflow for crawling, structuring, and processing datasets to facilitate model fine-tuning, alongside an ev

This project is a low-dependency engine designed for training large language models using native C and CUDA. It provides a bare-metal environment for tensor computation, allowing for the execution of neural network operations directly on hardware accelerators without the overhead of high-level software abstractions. The framework distinguishes itself by implementing manual gradient backpropagation and custom hardware-specific kernels, providing granular control over memory mapping and computational precision. It supports distributed training across multiple graphics processors and compute nod

TechEmpower FrameworkBenchmarks is an open-source project that provides a standardized, reproducible suite for measuring and comparing the performance of web frameworks across many languages. At its core, it defines a set of common server-side tasks—such as JSON serialization, database queries, and server-side template rendering—and executes them uniformly against hundreds of framework implementations to produce comparable throughput and latency metrics. The project is built around a multi-language benchmark harness that automates the full test lifecycle, from dependency installation and serve

Osim-rl is a research environment designed for the development and evaluation of reinforcement learning agents within physics-based musculoskeletal simulations. It provides a standardized interface that maps physiological state observations to muscle excitation control signals, enabling the study of human movement and biomechanics through iterative policy optimization. The framework distinguishes itself by integrating high-fidelity musculoskeletal modeling with tools for scientific benchmarking and reproducible experimentation. It allows researchers to define custom reward functions and adjus

This project is an LLM research orchestrator and autonomous AI agent framework designed to automate the scientific lifecycle. It functions as an end-to-end research pipeline and model training toolkit, managing everything from initial literature reviews and hypothesis testing to the final drafting of academic papers. The system is distinguished by its ability to convert unstructured academic PDFs into machine-executable knowledge layers, allowing agents to reproduce and extend research findings. It employs a two-loop orchestration architecture and a specialized research engineering skill libr

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

rllm is an asynchronous reinforcement learning framework for training language agents. It provides a unified pipeline that runs the same agent code for both evaluation and training, automatically capturing traces for gradient computation. The framework supports distributed reinforcement learning across multiple GPUs and nodes using pluggable backends, and executes agents in isolated sandboxes—either locally or in the cloud—for safe and scalable rollout collection. It trains agents built with LangGraph, SmolAgents, OpenAI Agents SDK, or custom frameworks without requiring core logic changes. T

SlowFast is a PyTorch video understanding framework and spatiotemporal neural network library. It serves as a toolset for video action recognition, enabling the training and evaluation of models designed to classify complex activities and objects within video sequences. The framework is distinguished by its use of dual-pathway spatiotemporal sampling to capture both slow and fast motions. It supports self-supervised video learning for pre-training models on unlabeled data and employs multigrid spatiotemporal training to optimize learning across multiple spatial and temporal resolutions. The

This project is a technical curriculum and development guide focused on large language model prompt engineering, fine-tuning, and the creation of retrieval augmented generation applications. It serves as a comprehensive resource for developers to master crafting precise instructions and textual patterns to improve the quality and predictability of model outputs. The material covers the end-to-end workflow of adapting open-source models to specific datasets and integrating language models with vector databases to generate responses based on private information. It also provides a systematic ap

SmolLM is a project dedicated to the development of small language models. It focuses on training and fine-tuning compact models that maintain high performance while utilizing fewer parameters. The project emphasizes efficient AI inference and on-device text generation, aiming to enable the deployment of lightweight models on edge devices with limited memory and processing power. It utilizes synthetic data generation to produce artificial datasets that improve the reasoning and training of these AI systems. The system supports a variety of optimization and training capabilities, including we

TCN is a deep learning sequence framework and library for building temporal convolutional networks. It provides a toolkit for implementing purely convolutional architectures to model sequential data as an alternative to recurrent neural networks. The project includes a sequence modeling benchmark suite designed to evaluate the accuracy and processing speed of architectures. This suite utilizes standardized tasks, including memory problems, digit classification, music, and language tasks, to quantify performance. The framework covers a range of structural components for sequence processing, s

MatchZoo is a deep learning framework designed for building, training, and evaluating neural networks that determine the relevance and similarity between pairs of textual inputs. It serves as a research platform for neural information retrieval, specifically supporting the development of models for document retrieval, question answering, and ranking tasks. The framework utilizes declarative architecture composition to define complex neural network structures. It includes automated hyper-parameter resolution to populate missing configuration parameters before model compilation and uses callbac