Awesome GitHub RepositoriesGPU Model Deployments

TensorRT este un motor de inferență pentru deep learning și un kit de dezvoltare software conceput pentru a optimiza și implementa rețele neuronale pentru execuție de înaltă performanță pe GPU-uri NVIDIA. Acesta funcționează ca un framework de accelerare GPU care reduce latența și crește debitul pentru modelele antrenate în timpul implementării în producție. Toolkit-ul importă modele din formatul Open Neural Network Exchange și le transformă în motoare optimizate. Utilizează optimizarea modelelor bazată pe grafuri, generarea de kernel-uri prin fuziunea straturilor și cuantizarea bazată pe precizie pentru a converti ponderile în virgulă mobilă în formate cu precizie mai mică. Framework-ul oferă capabilități pentru serializarea motoarelor specifice hardware-ului și suportă extinderea capabilităților de inferență prin plugin-uri personalizate pentru straturi specializate de rețele neuronale.

Acest proiect este un serviciu de embedding BERT de înaltă performanță și un server de inferență conceput pentru a mapa secvențele de text în vectori numerici de lungime fixă. Funcționează ca un microserviciu de învățare automată și server de model distribuit care decuplează gestionarea cererilor de calculul intensiv. Sistemul utilizează o infrastructură de mesagerie ZeroMQ pentru a oferi comunicare cu latență scăzută între clienții distribuiți și serverul de inferență. Încorporează procesarea în loturi pe partea de server și scalarea workload-ului GPU pentru a maximiza utilizarea hardware-ului și a gestiona volume mari de cereri. Platforma suportă infrastructura de căutare semantică prin generarea de embedding-uri cross-modale atât pentru text, cât și pentru imagini într-un spațiu vectorial partajat. Acest lucru permite căutarea cross-modală, clasarea relevanței conținutului și re-clasarea rezultatelor pe baza alinierii semantice între conținutul vizual și descrierile textuale. Serviciul poate fi implementat ca un microserviciu elastic accesibil prin protocoale gRPC, HTTP sau WebSocket, dispunând de streaming duplex non-blocant pentru gestionarea seturilor mari de date.

FlexLLMGen is an inference engine and runtime designed to run large language models on a single GPU by combining weight compression with tensor offloading. It reduces model weight memory usage by approximately 70% through 4-bit quantization, and stores model parameters, attention cache, and hidden states across GPU, CPU, and disk to fit models larger than available GPU memory. The project distinguishes itself through a throughput-oriented batching approach that processes multiple generation requests together in large batches to maximize throughput on a single GPU. It also supports distributed

FlexGen is an inference engine for large language models designed for high-throughput execution on single or multiple GPUs. It functions as a framework for managing model execution through a combination of memory offloading, weight compression, and pipeline orchestration. The system enables the execution of models that exceed available GPU memory by moving tensors and caches between GPU memory, system RAM, and disk storage. It utilizes 4-bit weight quantization to reduce the memory footprint of model parameters, allowing for increased batch processing capacity. The project covers distributed

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

This project is a PyTorch vision transformer framework designed for self-supervised learning. It implements a model that trains visual representations using a momentum teacher and self-distillation without the need for labeled data. The library functions as an image feature extractor and visual attention visualizer, allowing for the generation of high-dimensional vectors and the rendering of self-attention maps as heatmaps or videos to analyze model focus. It provides comprehensive tools for downstream vision evaluation, including linear probe classification, k-nearest neighbor categorizatio

mmagic is a multimodal training pipeline and framework for generative AI, focusing on visual synthesis and restoration. It provides the infrastructure to build and train models for tasks such as text-to-image and text-to-video generation, 3D-aware content synthesis, and high-fidelity image translation using diffusion models and generative adversarial networks. The project distinguishes itself through specialized capabilities for generative model personalization, including techniques for fine-tuning subjects and styles. It also supports advanced visual manipulations such as latent space interp

This project is a comprehensive educational curriculum and structured learning path covering the full lifecycle of large language models. It provides a guided progression through the theory, architecture, training, and deployment of these models. The curriculum includes specialized guides on transformer architecture, model training tutorials, and frameworks for designing autonomous agents. It also provides dedicated resources for studying model safety and ethics. The material covers a wide range of technical capabilities, including distributed training strategies, parameter-efficient fine-tu

OpenCompass is a comprehensive evaluation platform, benchmarking suite, and distributed model evaluator designed to measure the performance and accuracy of large language models. It provides a framework for benchmarking both open-source and API-based models against diverse datasets using standardized metrics and reproducible pipelines. The project features an automated judging framework that uses language models as judges to score and verify the quality of generated text. It includes a performance leaderboard system for comparing the relative capabilities of various models across industry-sta

OpenNMT-py is a PyTorch neural machine translation framework used for training and deploying neural machine translation and large language models. It functions as a distributed model training system, an inference engine, and a toolkit for fine-tuning large language models. The framework distinguishes itself with a dedicated toolkit for adapting large language models through low-rank adaptation, quantization, and instruction tuning. It also includes a neural machine translation server that allows trained models to be hosted and exposed via REST API endpoints. The project covers a broad range

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

This repository provides a collection of reference implementations, toolkits, and orchestration tools for training and deploying large-scale AI models on Cloud TPU hardware. It serves as a framework for managing the lifecycle of accelerator clusters, including hardware orchestration and the provisioning of high-performance compute infrastructure for machine learning workloads. The project specifically enables the pre-training of foundation models, large language models, and complex reasoning architectures through distributed training toolkits and multi-host scaling recipes. It further provide

Flash Linear Attention is a training framework and inference engine for sequence models that use linear attention and state space mechanisms, designed to process long contexts with reduced memory and compute overhead. It provides hardware-optimized token mixing layers and fused CUDA kernels that minimize memory bandwidth and launch overhead across different GPU architectures, and includes a causal inference engine that generates text token-by-token using cached hidden states for efficient autoregressive decoding. The project supports building hybrid sequence models that interleave standard at

cuml este o bibliotecă de machine learning accelerată pe GPU și un framework care utilizează CUDA pentru a accelera preprocesarea datelor tabelare și execuția modelelor. Oferă o suită de instrumente pentru antrenarea și implementarea modelelor de clasificare, regresie și clustering pe GPU-uri NVIDIA și clustere GPU. Biblioteca este concepută pentru scalabilitate, oferind un mediu de machine learning GPU distribuit care poate răspândi calculul și datele pe mai multe acceleratoare hardware și noduri pentru a gestiona seturi de date care depășesc memoria unui singur dispozitiv. Oglindește interfețele standard ale estimatorilor pentru a permite înlocuirea modelelor bazate pe CPU cu versiuni accelerate pe GPU în cadrul fluxurilor de lucru existente. Proiectul acoperă o gamă largă de capabilități de machine learning, incluzând învățarea supervizată, clustering-ul nesupervizat, căutarea celui mai apropiat vecin și reducerea dimensionalității de înaltă dimensiune. Include, de asemenea, preprocesarea datelor tabelare accelerată hardware pentru scalarea și codificarea caracteristicilor, extracția caracteristicilor textuale, analiza seriilor temporale și explicabilitatea predicțiilor modelului. Utilitarele de suport includ instrumente pentru generarea de seturi de date sintetice, serializarea stării modelului și calcularea metricilor de performanță ale modelului.