NVIDIA/triton-inference-server

Features

• Model Inference Servers - Acts as a dedicated server application hosting machine learning models to provide scalable, network-accessible inference services.
• Model Serving APIs - Provides a high-performance infrastructure for exposing deep learning models as network-accessible inference services.
• Batch Inference Engines - Groups individual requests into larger batches dynamically to increase total processing throughput.
• Chaining Pipelines - Enables the creation of complex workflows by chaining multiple models via ensembles or scripting.
• High Throughput Inference - Maximizes hardware utilization by combining dynamic batching and concurrent model execution for high-volume throughput.
• Model Deployment Pipelines - Provides standardized toolchains for deploying and serving AI models across cloud and edge infrastructure.
• Backend Runtimes - Provides pluggable runtime backends to execute models from various frameworks using hardware-optimized kernels.
• Model Inference Runtimes - Provides an execution layer supporting multiple AI frameworks to run models concurrently and optimize hardware use.
• Model Serving - Deploys trained machine learning models to production environments to provide scalable inference endpoints.
• Inference Batching - Implements dynamic batching to group individual inference requests, maximizing hardware throughput.
• Model Pipeline Orchestration - Provides ensemble pipelines that chain multiple models together into complex data processing sequences.
• In-Process Agent Execution - Allows the inference engine to be run directly within the host application process using native APIs.
• Inference Pipeline Orchestrators - Provides a framework for executing multi-stage machine learning inference pipelines using model ensembles.
• Edge AI Model Deployment - Optimizes and deploys machine learning models to run efficiently on embedded edge devices and data centers.
• Lifecycle Management - Controls active memory usage by dynamically loading and unloading models from a central repository.
• Inference Pipelines - Implements sequential chaining of models where the output of one serves as the input to the next.
• Model Lifecycle Management - Automatically loads or unloads models from a central storage repository based on configuration changes.
• Shared Memory Data Exchange - Utilizes shared memory regions for zero-copy transfer of large tensors between clients and the server.
• In-Process Integration - Exposes a native C-API allowing applications to embed the inference engine directly to eliminate network overhead.
• Execution Models - Executes multiple models or model instances concurrently to maximize hardware resource utilization.
• Concurrent Inference Instances - Manages multiple concurrent model instances in memory to process requests in parallel across GPUs and CPUs.
• Inference Performance Monitoring - Provides integrated telemetry for tracking GPU utilization, request throughput, and response latency.
• Infrastructure and Deployment - Optimized cloud inferencing solution for GPUs.

Star history