Awesome GitHub RepositoriesMulti-Framework Model Serving

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

هذا المشروع عبارة عن مورد تعليمي شامل ودليل تدريبي لبناء وتدريب ونشر نماذج تعلم الآلة باستخدام TensorFlow 2. يعمل كدليل تعليمي منظم يغطي مفاهيم التعلم العميق الأساسية، بما في ذلك معماريات الشبكات العصبية، والاشتقاق التلقائي، وعمليات الموترات (Tensors). يوفر الدليل توجيهات تقنية حول تحسين كفاءة التنفيذ من خلال إدارة ذاكرة GPU، والتدريب الموزع، وتكميم النماذج (Model Quantization). كما يتضمن أدلة مفصلة لبناء خطوط معالجة بيانات عالية الأداء وتصدير النماذج لخوادم الإنتاج، والأجهزة المحمولة، ومتصفحات الويب. تغطي المادة مجموعة واسعة من القدرات، بما في ذلك تطوير النماذج باستخدام الشبكات التلافيفية (CNN) والمتكررة (RNN)، وتنفيذ دوال خسارة وطبقات مخصصة، واستخدام النماذج المدربة مسبقاً للتعلم بنقل المعرفة (Transfer Learning). كما يتناول استراتيجيات النشر للأجهزة الطرفية (Edge Devices) واستخدام بيئات التشغيل السحابية لتسريع العتاد. تم تنفيذ المادة كمجموعة من دفاتر Jupyter Notebooks.

vllm-omni is a high-throughput serving engine and distributed inference framework designed for omni-modal models. It serves as a multi-modal model API server capable of generating text, image, video, and audio data, providing a standardized interface for remote client access. The system features a non-autoregressive generation engine for parallel media production and a robot policy inference server that acts as a real-time communication bridge to robotic hardware using specialized protocols. It supports hybrid execution models that combine sequential token generation with parallelized media g