Awesome GitHub RepositoriesTraining Pipelines

Nanochat is a lightweight execution environment designed for training and running language models on standard consumer hardware. It functions as both a neural network training framework and an inference engine, enabling users to perform backpropagation-based training and model execution directly on general-purpose processors without the need for dedicated graphics hardware. The project distinguishes itself through a suite of optimization tools that prioritize efficiency on local machines. By utilizing memory-mapped weight loading and CPU-optimized vector math, it maximizes throughput for inte

This project is a modular research toolkit designed for developing, training, and evaluating deep learning models for object detection, segmentation, and video instance tracking. It provides a flexible training engine that manages complex neural network execution, including distributed training, custom lifecycle hooks, and weight optimization. The framework is built around a hierarchical configuration system that allows users to define architectures, data pipelines, and training hyperparameters through composable, inheritable files. The project distinguishes itself through its highly modular

This project provides an end-to-end framework for adapting large language models to follow user instructions through supervised fine-tuning. It functions as a comprehensive training pipeline that enables the creation of specialized assistant models by minimizing the difference between predicted outputs and target responses within structured instruction datasets. The framework distinguishes itself by integrating synthetic data generation with memory-efficient training techniques. It utilizes powerful language models to iteratively expand small sets of human-written seeds into diverse, high-qua

This is a framework for training and sampling diffusion models to generate high-fidelity images, video, and 4D assets. It provides a modular environment for managing generative AI training pipelines, including the handling of datasets, noise sampling, and loss weighting to stabilize the creation of synthetic content. The project features a modular model configuration system that uses YAML-based assembly to define network submodules and conditioners. It also includes a dedicated toolset for AI image watermarking, allowing for the embedding and detection of invisible markers to verify the origi

This project is a generative speech synthesis engine that converts text into high-fidelity human speech. It utilizes a two-stage autoregressive transformer architecture that separates semantic token prediction from acoustic detail reconstruction to balance linguistic accuracy with audio quality. The system is designed to support multilingual output and conversational AI development, enabling the generation of context-aware speech that maintains flow across multiple dialogue turns. The platform distinguishes itself through a production-ready inference server that employs continuous batching to

Paddle is a deep learning framework designed for building, training, and deploying large-scale machine learning models. It incorporates a distributed training engine for optimizing performance across multiple chips and a model inference engine for transforming trained models into production-ready formats for cross-platform execution. The platform features a heterogeneous hardware abstraction and a standardized software stack that allows models to run across diverse hardware architectures through a common interface. It also includes a scientific computing library capable of solving complex dif

FinRL-Library is a reinforcement learning trading framework and algorithmic trading library used to develop and backtest automated financial trading strategies. It functions as a quantitative trading pipeline and financial market simulator, allowing users to build decision policies that optimize asset trading across various financial markets. The framework features a modular integration system for swapping reinforcement learning algorithms through a consistent API. It utilizes a standardized environment wrapper to encapsulate market dynamics into a state-action-reward interface, facilitating

PaddleDetection is an object detection framework designed for the end-to-end development, training, and deployment of computer vision models. It provides a comprehensive library of modular neural network architectures and pipelines that support object detection, instance segmentation, and multi-object tracking tasks. The project distinguishes itself through a configuration-driven approach that decouples model components like backbones and heads, allowing for the flexible assembly of custom vision workflows. It incorporates advanced techniques such as anchor-free detection logic, joint detecti

This is a PyTorch implementation of a text-to-image model designed for synthesizing high-fidelity images from natural language descriptions. It utilizes a diffusion image generator to transform latent embeddings into visual data through an iterative denoising process. The system employs a two-stage latent mapping process, using a CLIP-based latent prior to map text embeddings to image embeddings before decoding them into pixels. It features a cascading diffusion decoder that produces high-resolution imagery by passing low-resolution outputs through a sequence of models at increasing scales.

Nougat is a neural OCR system and LLM document parser designed to convert images of academic PDF documents into structured markdown text and mathematical formulas. It functions as a PDF to markdown converter that uses deep learning to handle layout and formula recognition. The project provides a document training pipeline for generating datasets and training neural networks to recognize specific academic document styles. This includes utilities for training dataset generation, neural model training, and model checkpoint management to ensure reproducible deployment. The system covers a broad

LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

nnU-Net is a PyTorch-based deep learning framework for the supervised semantic segmentation of 2D and 3D biomedical images. It functions as an automated medical imaging pipeline that generates predicted masks and labels from clinical images. The system distinguishes itself by using dataset-driven auto-configuration to automatically select the optimal network architecture, preprocessing steps, and training hyperparameters based on the specific properties of the input medical dataset. The framework covers a broad range of capabilities including medical dataset preparation, intensity normalizat

Gluon-CV este o bibliotecă de computer vision pentru MXNet care oferă o colecție cuprinzătoare de arhitecturi de viziune pre-implementate și pipeline-uri de antrenament. Servește drept toolkit de cercetare în deep learning și o grădină zoologică de modele (model zoo) care conține ponderi pre-antrenate de ultimă generație pentru analiza imaginilor și a videoclipurilor. Proiectul include o bibliotecă specializată de estimare a posturii umane și un toolkit de compresie a modelelor. Aceste instrumente permit tăierea (pruning) și cuantizarea modelelor de deep learning pentru a crește viteza de inferență și a facilita implementarea pe hardware edge cu resurse limitate. Biblioteca acoperă o gamă largă de capabilități de viziune, inclusiv clasificarea imaginilor, detectarea obiectelor și segmentarea semantică și de instanță. De asemenea, oferă instrumente pentru analiza video, cum ar fi recunoașterea acțiunilor, urmărirea obiectelor și estimarea adâncimii monoculare. Antrenamentul este susținut prin pipeline-uri automatizate și sarcini de lucru distribuite multi-GPU pentru a accelera convergența modelului.

rllm is an asynchronous reinforcement learning framework for training language agents. It provides a unified pipeline that runs the same agent code for both evaluation and training, automatically capturing traces for gradient computation. The framework supports distributed reinforcement learning across multiple GPUs and nodes using pluggable backends, and executes agents in isolated sandboxes—either locally or in the cloud—for safe and scalable rollout collection. It trains agents built with LangGraph, SmolAgents, OpenAI Agents SDK, or custom frameworks without requiring core logic changes. T

OpenChat is a conversational AI agent builder and customer service automation platform that uses large language models to power customer support chatbots across multiple channels. It provides tools for defining AI agent behavior, training on custom knowledge, managing actions, and controlling autopilot responses per channel. The platform enables deploying AI agents on web, phone, email, SMS, and WhatsApp, with a unified inbox for managing conversations across all channels. It includes CRM synchronization, automated workflows, contact segmentation, and analytics for tracking customer satisfact

AdalFlow este un framework de agenți AI autonomi și o bibliotecă de aplicații LLM concepută pentru construirea de fluxuri de lucru modulare. Servește ca interfață agnostic-model și orchestrator de pipeline-uri RAG, permițând utilizatorilor să dezvolte agenți ReAct care utilizează raționamentul iterativ și execuția de instrumente externe pentru a rezolva sarcini complexe. Proiectul se distinge printr-un sistem de optimizare a prompt-urilor care utilizează gradient descent textual pentru a rafina automat template-urile de prompt și exemplele few-shot. Acesta tratează feedback-ul modelului ca pe un semnal diferențiabil, permițând o formă de backpropagation pentru LLM-uri pentru a îmbunătăți iterativ calitatea output-ului pe baza metricilor de evaluare. Framework-ul acoperă o suprafață largă de capabilități, inclusiv retrieval-augmented generation cu căutare semantică vectorială și reranking, urmărirea execuției bazată pe span pentru observabilitate și parsare structurată bazată pe schemă. Oferă un strat de comunicare unificat pentru numeroși furnizori de modele proprietare și open-source și suportă conversia funcțiilor Python în interfețe standardizate de instrumente. Sistemul este implementat în Python și se integrează cu MLflow pentru urmărirea și analiza fluxurilor de lucru.