Awesome GitHub RepositoriesTraining Pipelines

Nanochat is a lightweight execution environment designed for training and running language models on standard consumer hardware. It functions as both a neural network training framework and an inference engine, enabling users to perform backpropagation-based training and model execution directly on general-purpose processors without the need for dedicated graphics hardware. The project distinguishes itself through a suite of optimization tools that prioritize efficiency on local machines. By utilizing memory-mapped weight loading and CPU-optimized vector math, it maximizes throughput for inte

This project is a modular research toolkit designed for developing, training, and evaluating deep learning models for object detection, segmentation, and video instance tracking. It provides a flexible training engine that manages complex neural network execution, including distributed training, custom lifecycle hooks, and weight optimization. The framework is built around a hierarchical configuration system that allows users to define architectures, data pipelines, and training hyperparameters through composable, inheritable files. The project distinguishes itself through its highly modular

This project provides an end-to-end framework for adapting large language models to follow user instructions through supervised fine-tuning. It functions as a comprehensive training pipeline that enables the creation of specialized assistant models by minimizing the difference between predicted outputs and target responses within structured instruction datasets. The framework distinguishes itself by integrating synthetic data generation with memory-efficient training techniques. It utilizes powerful language models to iteratively expand small sets of human-written seeds into diverse, high-qua

This is a framework for training and sampling diffusion models to generate high-fidelity images, video, and 4D assets. It provides a modular environment for managing generative AI training pipelines, including the handling of datasets, noise sampling, and loss weighting to stabilize the creation of synthetic content. The project features a modular model configuration system that uses YAML-based assembly to define network submodules and conditioners. It also includes a dedicated toolset for AI image watermarking, allowing for the embedding and detection of invisible markers to verify the origi

This project is a generative speech synthesis engine that converts text into high-fidelity human speech. It utilizes a two-stage autoregressive transformer architecture that separates semantic token prediction from acoustic detail reconstruction to balance linguistic accuracy with audio quality. The system is designed to support multilingual output and conversational AI development, enabling the generation of context-aware speech that maintains flow across multiple dialogue turns. The platform distinguishes itself through a production-ready inference server that employs continuous batching to

Paddle is a deep learning framework designed for building, training, and deploying large-scale machine learning models. It incorporates a distributed training engine for optimizing performance across multiple chips and a model inference engine for transforming trained models into production-ready formats for cross-platform execution. The platform features a heterogeneous hardware abstraction and a standardized software stack that allows models to run across diverse hardware architectures through a common interface. It also includes a scientific computing library capable of solving complex dif

FinRL-Library is a reinforcement learning trading framework and algorithmic trading library used to develop and backtest automated financial trading strategies. It functions as a quantitative trading pipeline and financial market simulator, allowing users to build decision policies that optimize asset trading across various financial markets. The framework features a modular integration system for swapping reinforcement learning algorithms through a consistent API. It utilizes a standardized environment wrapper to encapsulate market dynamics into a state-action-reward interface, facilitating

PaddleDetection is an object detection framework designed for the end-to-end development, training, and deployment of computer vision models. It provides a comprehensive library of modular neural network architectures and pipelines that support object detection, instance segmentation, and multi-object tracking tasks. The project distinguishes itself through a configuration-driven approach that decouples model components like backbones and heads, allowing for the flexible assembly of custom vision workflows. It incorporates advanced techniques such as anchor-free detection logic, joint detecti

This is a PyTorch implementation of a text-to-image model designed for synthesizing high-fidelity images from natural language descriptions. It utilizes a diffusion image generator to transform latent embeddings into visual data through an iterative denoising process. The system employs a two-stage latent mapping process, using a CLIP-based latent prior to map text embeddings to image embeddings before decoding them into pixels. It features a cascading diffusion decoder that produces high-resolution imagery by passing low-resolution outputs through a sequence of models at increasing scales.

Nougat is a neural OCR system and LLM document parser designed to convert images of academic PDF documents into structured markdown text and mathematical formulas. It functions as a PDF to markdown converter that uses deep learning to handle layout and formula recognition. The project provides a document training pipeline for generating datasets and training neural networks to recognize specific academic document styles. This includes utilities for training dataset generation, neural model training, and model checkpoint management to ensure reproducible deployment. The system covers a broad

LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

nnU-Net is a PyTorch-based deep learning framework for the supervised semantic segmentation of 2D and 3D biomedical images. It functions as an automated medical imaging pipeline that generates predicted masks and labels from clinical images. The system distinguishes itself by using dataset-driven auto-configuration to automatically select the optimal network architecture, preprocessing steps, and training hyperparameters based on the specific properties of the input medical dataset. The framework covers a broad range of capabilities including medical dataset preparation, intensity normalizat

Gluon-CV هي مكتبة رؤية حاسوبية لـ MXNet توفر مجموعة شاملة من معماريات الرؤية وخطوط أنابيب التدريب المنفذة مسبقاً. تعمل كمجموعة أدوات لأبحاث التعلم العميق وحديقة نماذج تحتوي على أوزان مدربة مسبقاً ومتطورة لتحليل الصور والفيديو. يتضمن المشروع مكتبة متخصصة لتقدير وضعية الإنسان ومجموعة أدوات لضغط النماذج. تسمح هذه الأدوات بتقليم وتكميم نماذج التعلم العميق لزيادة سرعة الاستدلال وتسهيل النشر على أجهزة الحافة المقيدة. تغطي المكتبة مجموعة واسعة من قدرات الرؤية، بما في ذلك تصنيف الصور، واكتشاف الكائنات، والتجزئة الدلالية والمثالية. كما توفر أدوات لتحليل الفيديو، مثل التعرف على الإجراءات، وتتبع الكائنات، وتقدير العمق أحادي العين. يتم دعم التدريب من خلال خطوط أنابيب مؤتمتة وأحمال عمل موزعة على وحدات GPU متعددة لتسريع تقارب النموذج.

rllm is an asynchronous reinforcement learning framework for training language agents. It provides a unified pipeline that runs the same agent code for both evaluation and training, automatically capturing traces for gradient computation. The framework supports distributed reinforcement learning across multiple GPUs and nodes using pluggable backends, and executes agents in isolated sandboxes—either locally or in the cloud—for safe and scalable rollout collection. It trains agents built with LangGraph, SmolAgents, OpenAI Agents SDK, or custom frameworks without requiring core logic changes. T

OpenChat is a conversational AI agent builder and customer service automation platform that uses large language models to power customer support chatbots across multiple channels. It provides tools for defining AI agent behavior, training on custom knowledge, managing actions, and controlling autopilot responses per channel. The platform enables deploying AI agents on web, phone, email, SMS, and WhatsApp, with a unified inbox for managing conversations across all channels. It includes CRM synchronization, automated workflows, contact segmentation, and analytics for tracking customer satisfact

AdalFlow هو إطار عمل لوكلاء الذكاء الاصطناعي المستقلين ومكتبة تطبيقات للنماذج اللغوية الكبيرة (LLM) مصممة لبناء سير عمل معياري. يعمل كواجهة محايدة للنموذج ومنسق لخط أنابيب RAG، مما يسمح للمستخدمين بتطوير وكلاء ReAct يستخدمون التفكير التكراري وتنفيذ الأدوات الخارجية لحل المهام المعقدة. يتميز المشروع بنظام تحسين المطالبة (prompt optimization) الذي يستخدم الانحدار المتدرج النصي لتحسين قوالب المطالبة وأمثلة التعلم القليل (few-shot) تلقائياً. يعامل ملاحظات النموذج كإشارة قابلة للاشتقاق، مما يتيح شكلاً من أشكال الانتشار العكسي للنماذج اللغوية الكبيرة لتحسين جودة المخرجات تكرارياً بناءً على مقاييس التقييم. يغطي إطار العمل سطح قدرات واسعاً، بما في ذلك التوليد المعزز بالاسترجاع (RAG) مع البحث الدلالي المتجه وإعادة الترتيب، وتتبع التنفيذ القائم على النطاق للمراقبة، والتحليل الهيكلي القائم على المخطط. يوفر طبقة اتصال موحدة للعديد من مزودي النماذج المملوكة والمفتوحة المصدر ويدعم تحويل وظائف Python إلى واجهات أدوات قياسية. تم تنفيذ النظام بلغة Python ويتكامل مع MLflow لتتبع وتحليل سير العمل.