Open-source alternatives to Datasets

Easy-dataset is a comprehensive platform designed for the end-to-end management of machine learning datasets, specifically tailored for language and vision model fine-tuning. It functions as a centralized environment for the entire data lifecycle, encompassing the automated generation of synthetic training data, the structural organization of document collections, and the systematic annotation of individual data points. The platform distinguishes itself through its integrated evaluation and orchestration capabilities. It provides a dedicated suite for benchmarking models, featuring blind side

RecBole is a PyTorch-based recommendation framework designed for building, training, and evaluating a wide variety of recommendation algorithms. It serves as a standardized benchmark environment that allows for the comparison of different model architectures using public datasets and consistent evaluation metrics. The project provides specialized toolkits for sequential recommendation and knowledge-graph integration, enabling the prediction of item sequences based on user history or the incorporation of structured external knowledge. It includes a dedicated hyperparameter optimization engine

Lightly is a self-supervised learning framework and computer vision data curation tool designed to manage large image datasets and train models on unlabeled data. It functions as a PyTorch vision library and dataset management SDK, providing tools to convert raw images into high-dimensional vectors for similarity search, visualization, and feature extraction. The project implements a variety of self-supervised architectures, including MoCo, SimCLR, VICReg, Barlow Twins, and masked image modeling. It distinguishes itself by combining these learning frameworks with active learning capabilities,

PointNet++ is a deep learning framework designed for processing and classifying 3D point cloud data. It utilizes a hierarchical feature learning architecture to extract geometric patterns from sampled 3D point sets. The framework implements a variety of 3D analysis tools, including a point cloud classifier for categorizing objects based on spatial coordinates and surface normals, a semantic scene segmenter for labeling surfaces in large-scale environments, and a tool for 3D object part segmentation. The system covers a broad range of capabilities including geometric feature extraction, 3D da

Apache Spark is a unified distributed data processing engine designed for large-scale data analysis and computation graphs. It functions as a distributed machine learning framework, a graph processing system, a real-time stream processor, and a SQL analytics engine. The system enables the execution of distributed SQL querying, large-scale graph analysis, and real-time stream analytics across clusters of machines. It also provides a scalable environment for implementing machine learning algorithms and predictive model development on massive datasets. The engine incorporates relational query e

This project is a community-maintained, open-access directory of high-quality public datasets. It serves as a centralized reference point for researchers, developers, and data scientists to locate reliable information sources across a wide spectrum of industries and scientific fields. By providing a structured index, the repository facilitates the discovery of data necessary for exploratory analysis, machine learning model training, and the development of data-intensive applications. The directory distinguishes itself through a lightweight, platform-agnostic approach to resource indexing that

mmaction2 is a PyTorch video understanding toolbox designed for training and evaluating deep learning models. It serves as a framework for action recognition, temporal localization, and spatio-temporal action detection, providing specialized tools for both pixel-based video analysis and skeleton-based action recognition. The project distinguishes itself through a modular architecture featuring registry-based component discovery and hierarchical, config-driven model assembly. It supports multi-modal feature fusion, integrating RGB frames, optical flow, and audio, and includes capabilities for

Tensorpack is a high-performance TensorFlow training framework and distributed deep learning toolkit. It provides a suite of tools for building and training neural networks with a focus on execution speed and architectural flexibility. The project serves as a neural network optimization suite, implementing high-efficiency execution patterns to reduce training overhead. It functions as a parallel data loading pipeline, using automated parallelization to maximize throughput when processing large datasets. The toolkit covers distributed training across multiple GPUs and compute clusters using d

Pentaho Kettle is an enterprise ETL data integration platform designed to extract, transform, and load data between disparate sources and target databases. It functions as a metadata-driven orchestrator that utilizes a visual workflow designer to create and manage complex sequences of data tasks and transformation pipelines. The system is distinguished by its distributed data processing engine, which executes workloads across clusters of server nodes to increase throughput. It employs a plugin-based architecture, allowing the platform to be extended via external JAR files to provide connectiv

This project is a collection of foundational machine learning algorithms and data science tools implemented in Python. It focuses on building the logic of these tools using basic programming primitives rather than relying on specialized libraries. The implementation covers several core domains, including a linear algebra library for matrix and vector operations, a statistical analysis toolkit for probability and hypothesis testing, and a framework for map-reduce distributed processing. It also includes implementations for natural language processing, graph theory for network analysis, and var

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

zero_nlp is a distributed framework for training and fine-tuning large language models and multimodal architectures. It provides a specialized toolkit for distributed model parallelism, allowing neural network layers and weights to be partitioned across multiple GPU devices to train models that exceed the memory capacity of a single processor. The project distinguishes itself through a combination of high-throughput data pipelines and parameter-efficient tuning. It utilizes multi-threading and memory mapping to preprocess and stream datasets exceeding 100GB and implements memory-saving adapta

This project is an interactive, web-based notebook environment designed for distributed data science and large-scale computing. It serves as a development tool for executing code and performing data analysis specifically within the Apache Spark framework, providing a browser-based interface that combines code execution with reactive data visualization. The platform distinguishes itself through its deep integration with distributed infrastructure, allowing users to manage cluster resources, configure runtime dependencies, and isolate execution processes for individual notebooks. It supports co

This project is a comprehensive collection of educational examples and reference implementations for building vision and language models using PyTorch. It serves as a deep learning tutorial covering the end-to-end process of developing neural networks, from initial architecture definition to final production deployment. The repository provides detailed guides on implementing a wide range of domain-specific models, including convolutional neural networks for object detection and segmentation, as well as transformer and recurrent architectures for natural language processing. It emphasizes gene

NVIDIA DALI is a GPU-accelerated data loading and preprocessing library designed for deep learning workflows. It constructs high-performance data pipelines that offload decoding, augmentation, and normalization to the GPU, eliminating CPU bottlenecks in training and inference. The library reads data from multiple storage formats and streams it directly into GPU memory, with support for multi-GPU execution to scale throughput across large-scale workloads. DALI distinguishes itself by enabling data pipelines to be built once and executed across multiple deep learning frameworks without code cha

AIOS is an LLM agent operating system and orchestration kernel designed to manage memory, resource scheduling, and tool execution for multiple autonomous AI agents. It serves as a comprehensive framework for developing and deploying agents, featuring a dedicated resource manager that coordinates model backends, GPU memory, and isolated kernel instances. The system distinguishes itself through a semantic memory engine that uses vector search and autonomous clustering for long-term knowledge management, and a semantic file system that allows users to control computer files and system operations

AllenNLP is a PyTorch-based research library and deep learning language toolkit designed for developing and training neural network architectures for linguistic tasks. It provides a distributed training system that coordinates data and gradients across multiple GPUs and a framework for integrating pretrained transformer architectures. The system distinguishes itself with a dedicated algorithmic bias mitigation tool used to identify and reduce bias in linguistic model predictions. It also includes model influence analysis to interpret predictions by calculating the influence of specific traini

LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

CVAT is an open-source computer vision annotation tool and visual dataset management platform. It provides a self-hosted interface for labeling images, videos, and 3D data to create datasets for vision AI models. The platform features AI-assisted data labeling to automate the creation of masks and bounding boxes, utilizing a plug-in system to connect external machine learning models. It includes a consensus-based quality assurance system that verifies label accuracy by comparing independent annotations. The system covers collaborative team management, project organization through task decomp

Ionicons is a library of hand-crafted scalable vector icons and an interface design asset pack used to build consistent user interfaces across web and mobile applications. It provides a collection of vector graphics designed for design system implementation and web component icon systems. The library includes a cross-platform icon set that automatically adapts its style based on the user's operating system to maintain a native look and feel. It also supports the integration of custom SVG assets through remote URL resolution to extend the available set of visuals. The system covers SVG asset

This is an interactive notebook-based course that teaches machine learning from Python fundamentals through deep learning and natural language processing. It uses real datasets and multiple frameworks within a structured, hands-on curriculum that combines concise explanations with executable code cells, built-in datasets, and embedded exercise checkpoints. Learning progresses through data preparation and exploration, classical machine learning workflows, computer vision with convolutional neural networks, and natural language processing with deep learning, all delivered as a cohesive progressi

Acra is an Android crash reporting framework and diagnostic data collector designed to detect failures and capture device diagnostics, system logs, and application state. It serves as an embeddable library for capturing and processing crash reports, providing a pipeline to send this data to custom backends or via email. The project features a plugin-based sender architecture that allows reports to be routed through HTTP endpoints, email clients, or proprietary backend implementations. It includes a user-controlled reporting system with preference toggles and interactive dialogs to manage user

Cookbook is a comprehensive knowledge base and reference repository for data engineering. It serves as a centralized directory for data architecture patterns, professional career roadmaps, and a curated collection of public datasets. The project provides a structured guide for transitioning into specialized data engineering roles through skill-matrix mapping and technical interview preparation. It further distinguishes itself by documenting real-world industry case studies and decomposing large-scale industrial implementations into repeatable architectural patterns. The repository covers a b

This project is a comprehensive collection of machine learning educational resources, featuring a Python-based curriculum, study guides for deep learning, and a specialized knowledge base for machine learning operations. It provides structured learning paths that guide users from foundational programming through to advanced neural network implementations. The repository focuses on interactive learning by providing a directory of executable notebooks and cloud-hosted experiments. It maps theoretical research papers and textbooks to practical code implementations and maintains a curated directo

This repository contains programming assignments and lecture notes from Andrew Ng's foundational deep learning course specialization on Coursera. The materials cover core neural network training techniques including optimization algorithms, normalization methods, regularization approaches, parameter initialization strategies, and learning rate scheduling to improve model convergence and generalization. The coursework explores design principles where successive neural network layers learn progressively more abstract feature representations from input data. It provides guidance on selecting ope

GluonTS is a probabilistic time series library and deep learning forecasting framework. It provides a toolkit for building, training, and evaluating neural network architectures that predict future values as probability distributions to quantify uncertainty. The project distinguishes itself by supporting zero-shot forecasting and integrating diverse modeling approaches, including deep probabilistic neural networks and wrappers for external statistical libraries such as Prophet and R forecast. It implements specialized architectural primitives like causal convolutions and invertible residual n

This is a high-performance image component and native caching library for mobile applications. It provides tools for image optimization and asset preloading to reduce visual flicker and increase loading speeds. The project differentiates itself through a mobile asset preloader that downloads remote images into a local cache before they are requested. It also includes a system for assigning loading priority levels to ensure critical visual assets are fetched first and supports remote asset authentication by attaching custom HTTP headers to requests. The library covers a broad range of capabil