Awesome GitHub RepositoriesDataset Versioning Platforms

Datasets is a library designed for the management, processing, and sharing of large-scale data collections for machine learning workflows. It functions as both a data processing framework and a versioning platform, providing tools to organize, filter, and transform massive datasets while ensuring reproducibility across research and development teams. The library distinguishes itself by enabling the handling of datasets that exceed available system memory. It utilizes memory-mapped file access, disk-based caching, and lazy iterative streaming to maintain performance when working with large-sca

Mastra is an orchestration framework designed for building, deploying, and managing autonomous AI agents and multi-agent systems. It provides a comprehensive suite of primitives for creating resilient AI applications, including durable workflow orchestration, event-driven agent loops, and semantic memory management. By integrating these core components, the platform enables developers to build complex, multi-step processes that can reason about goals and execute tasks without manual intervention. The framework distinguishes itself through its focus on observability and secure, isolated execut

Opik is an observability and evaluation platform designed for generative AI applications and agentic workflows. It provides a centralized environment for tracing execution flows, managing prompt templates, and monitoring production performance, allowing teams to gain visibility into complex model interactions and tool usage without requiring manual application code changes. The platform distinguishes itself through its integrated approach to the AI development lifecycle, combining distributed trace instrumentation with automated evaluation frameworks. It supports model-as-a-judge scoring, syn

DVC is a data versioning tool and pipeline orchestrator designed to track large datasets and machine learning models. It functions as a system for managing large data artifacts by storing lightweight metadata in version control while keeping the actual binaries in a separate cache. The project serves as an experiment tracker and remote storage synchronizer, enabling the execution and comparison of machine learning iterations based on hyperparameters and performance metrics. It provides a bridge for pushing and pulling these large data artifacts between local environments and cloud or on-premi

DVC is a data versioning tool and pipeline orchestrator designed to track large datasets and machine learning models using external storage and metadata pointers. It integrates with Git by utilizing placeholders to keep heavy artifacts out of the repository while maintaining a versioned link between code and data. The system manages remote data caches through a synchronization layer that connects local environments to cloud storage or network filesystems. It also functions as an experiment tracker, recording hyperparameters and metrics to compare the performance of different model iterations.

This library provides a diagnostic toolkit for automated data profiling and exploratory analysis. It generates comprehensive statistical summaries and visual reports for tabular datasets, enabling users to identify distribution patterns, missing values, and quality anomalies through a unified interface. The project distinguishes itself by offering differential analysis, which allows for the comparison of two dataset versions to track structural and statistical changes over time. It supports large-scale data processing through lazy evaluation and provides interactive widgets that embed directl

Kedro is a data science pipeline framework and orchestration tool designed to build reproducible and modular data engineering workflows. It functions as an MLOps project template and Python data workflow tool that enforces software engineering best practices to move projects from prototype to production. The system distinguishes itself through a centralized data catalog manager that abstracts data access and versioning across various file formats and cloud storage systems. It further separates processing logic from data access via a lazy-loading data registry and provides a standardized proje

Wandb is a centralized platform for machine learning experiment tracking, model registry management, and workflow orchestration. It provides a comprehensive suite of tools for logging, visualizing, and versioning training metrics, model artifacts, and hyperparameter sweeps to ensure reproducibility across development cycles. The platform also functions as an observability tool for large language model applications, enabling the tracing of execution steps, token usage, and reasoning processes. The project distinguishes itself through its event-driven automation capabilities, which allow users

PyCaret is a Python AutoML platform and MLOps lifecycle manager designed to automate machine learning workflows. It functions as a low-code environment that leverages a scikit-learn native engine to execute preprocessing, training, and evaluation for tabular data. The platform distinguishes itself as an LLM-powered ML copilot, using large language model agents to analyze datasets, design experiment configurations, and explain model results. It also serves as a Kubernetes ML orchestrator and model registry, enabling the versioning of trained pipelines and their promotion to production API endp

LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

Oumi is a comprehensive large language model development platform designed for synthesizing data, fine-tuning models, and running performance evaluations. It serves as a unified environment for the entire model lifecycle, encompassing a training and fine-tuning suite, an evaluation framework, and tools for synthetic data generation and model distillation. The platform is distinguished by its iterative, failure-driven synthesis approach, which analyzes model weaknesses during evaluation to generate targeted training data. It utilizes an LLM-based judge framework to programmatically score respo

ClearML is a comprehensive MLOps platform designed to manage the end-to-end machine learning lifecycle, from initial experimentation to production deployment. It provides a suite of integrated tools including a pipeline orchestrator for automating workflows, an experiment tracking tool for logging hyperparameters and metrics, and a metadata-driven data versioning system for managing large-scale datasets and model artifacts. The platform is distinguished by its advanced compute management and serving capabilities. It features a GPU compute manager that supports fractional resource slicing and

Pachyderm is a containerized, versioned, and lineage-tracked data pipeline platform that runs natively on Kubernetes. It combines a distributed file system backend with immutable data versioning, so every commit to a data repository creates an auditable snapshot, and every pipeline step executes as an isolated container. The platform is defined by a data-centric pipeline model where pipelines are specified by their input and output data repositories rather than explicit task sequences, and provenance is recorded as a directed acyclic graph of commits linking output data to its input sources an

lakeFS هو نظام إصدارات لبحيرات البيانات يوفر تفرعاً (branching) والتزامات (commits) تشبه Git لمجموعات البيانات الكبيرة المخزنة في تخزين الكائنات. يعمل كطبقة تحكم في الإصدار، مما يتيح إنشاء لقطات غير قابلة للتغيير، والتزامات ذرية، وتفرعاً بدون نسخ (zero-copy) لإنشاء بيئات معزولة لتجارب البيانات دون تكرار الملفات الفيزيائية. يعمل النظام كبوابة تخزين متوافقة مع S3 وفهرس Iceberg REST، مما يسمح لبروتوكولات التخزين السحابي القياسية والعملاء المتوافقين بإدارة الجداول ذات الإصدارات. يعمل كحارس لجودة البيانات باستخدام نظام خطافات (hooks) قائم على الأحداث للتحقق من مجموعات البيانات مقابل سياسات الحوكمة قبل دمج التغييرات في الإنتاج. تغطي المنصة قدرات واسعة لحوكمة البيانات، بما في ذلك التعاون عبر طلبات السحب (pull requests)، والتحكم في الوصول القائم على الأدوار، وتتبع أصل البيانات. يوفر تكاملاً لتنسيق سير العمل، وخطوط أنابيب التعلم الآلي، ومحركات حوسبة البيانات الضخمة المختلفة، ويدعم اتصال التخزين متعدد السحابة ومزامنة الهوية عبر SSO وSCIM. يمكن تثبيت البرنامج باستخدام ملفات ثنائية، أو حاويات، أو Helm charts للنشر على Kubernetes.

CML هي أداة لأتمتة خطوط الأنابيب لتدريب وتقييم نماذج تعلم الآلة، وتعمل كنظام CI/CD لتعلم الآلة. تعمل كمنسق للحوسبة السحابية ومدير سير عمل يعتمد على Git يقوم بأتمتة دورات تدريب النماذج من خلال إدارة الفروع، والالتزامات (commits) المؤتمتة، والتقارير المتكاملة. يتميز المشروع بتوفير نسخ سحابية مؤقتة أو عقد Kubernetes لتوفير أجهزة متخصصة للمهام كثيفة الحوسبة. كما يدير مشغلات الحوسبة عن بُعد، مما يسمح بربط مجموعات GPU ذاتية الاستضافة أو أجهزة محلية لتنفيذ سير عمل تعلم الآلة المحاوي (containerized). يغطي النظام مجموعة واسعة من الإمكانيات بما في ذلك تتبع تجارب تعلم الآلة، حيث يتم نشر مقاييس الأداء والتصورات مباشرة في طلبات السحب (pull requests) الخاصة بالتحكم في الإصدار. يتعامل مع أتمتة خط أنابيب تعلم الآلة من استيراد البيانات الأولي وإصدارها إلى إنشاء تقارير سير العمل المنسقة وروابط التصور الخارجية. توفر الأداة فائدة إضافية لإدارة البنية التحتية من خلال تصحيح الأخطاء عن بُعد عبر SSH والقدرة على استئناف المهام التي تمت مقاطعتها.