Awesome GitHub RepositoriesMulti-Framework Model Serving

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

Este proyecto es un recurso educativo integral y un manual de tutoriales para construir, entrenar y desplegar modelos de machine learning usando TensorFlow 2. Sirve como una guía de aprendizaje estructurada que cubre conceptos fundamentales de deep learning, incluyendo arquitecturas de redes neuronales, diferenciación automática y operaciones con tensores. El manual proporciona orientación técnica sobre cómo optimizar la eficiencia de ejecución mediante la gestión de memoria de GPU, entrenamiento distribuido y cuantización de modelos. También incluye guías detalladas para construir pipelines de datos de alto rendimiento y exportar modelos para servidores de producción, dispositivos móviles y navegadores web. El material abarca una amplia gama de capacidades, incluyendo el desarrollo de modelos con redes convolucionales y recurrentes, la implementación de funciones de pérdida y capas personalizadas, y el uso de modelos preentrenados para transfer learning. También aborda estrategias de despliegue para dispositivos edge y el uso de entornos de ejecución en la nube para aceleración por hardware. El recurso está implementado como una colección de Jupyter Notebooks.

vllm-omni is a high-throughput serving engine and distributed inference framework designed for omni-modal models. It serves as a multi-modal model API server capable of generating text, image, video, and audio data, providing a standardized interface for remote client access. The system features a non-autoregressive generation engine for parallel media production and a robot policy inference server that acts as a real-time communication bridge to robotic hardware using specialized protocols. It supports hybrid execution models that combine sequential token generation with parallelized media g