triton-inference-server/server
10,768 星标·1,796 分支·Python·BSD-3-Clause·8 次浏览docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html↗
Server
Triton Inference Server is a high-performance server designed to deploy machine learning models from multiple frameworks across GPUs and CPUs. It functions as a hardware-accelerated inference engine and a gRPC inference gateway, providing a standardized communication layer for transmitting binary tensor data with low latency.
The system acts as a multi-framework model orchestrator, allowing users to link multiple AI models into ensembles and scripts to create complex inference pipelines. It also serves as a model lifecycle manager, providing controls to load, unload, and monitor the performance of models in production environments.
Throughput is optimized via dynamic batching, concurrent model execution, and stateful sequence batching. The server supports extensibility through custom inference backends implemented in C++ or Python and utilizes shared memory communication to reduce data copying overhead.
Observability is provided through performance monitoring of hardware utilization, request throughput, and response latency.
Features
- Model Serving - Serves as a high-performance server that deploys models from various deep learning frameworks across GPUs and CPUs.
- Hardware-Accelerated Inference - Optimizes throughput using dynamic batching and concurrent execution on specialized hardware accelerators.
- Inference Pipeline Orchestrators - Provides a system for linking multiple models into ensembles and scripts to create complex, multi-stage inference pipelines.
- High-Throughput Model Serving - Optimizes model performance using dynamic batching and concurrent execution to handle large volumes of requests.
- Inference API Servers - Exposes standardized HTTP and gRPC endpoints to allow clients to submit data and receive model predictions.
- Model Inference Servers - Provides a high-performance server that deploys machine learning models from multiple frameworks via HTTP and gRPC.
- Inference Optimizations - Increases efficiency through concurrent model execution, dynamic batching, and stateful sequence batching to maximize throughput.
- Model Gateways - Functions as a standardized communication gateway for transmitting binary tensor data to models with low latency.
- Inference Backends - Allows the implementation of custom backends and processing operations to support new machine learning frameworks.
- Lifecycle Management - Implements controls for loading and unloading models to optimize memory and resource usage during production serving.
- Custom Backend SDKs - Extends inference capabilities by allowing the implementation of custom C++ or Python backends.
- Backend Runtimes - Decouples core server logic from framework-specific runtimes using separate processes or libraries for model execution.
- Model Lifecycle Managers - Provides a resource controller for loading, unloading, and monitoring the performance of AI models in production.
- Inference Pipelines - Sequentially chains multiple models together by routing the output of one model as the input to the next.
- Stateful Sequence Batching - Tracks the state of long-running requests across multiple calls to handle sequential data like text or audio.
- Shared Memory Data Exchange - Transfers large tensor data between client and server processes using memory-mapped files to eliminate data copying.
- Inference Batching - Groups individual inference requests into larger batches at runtime to maximize hardware utilization and throughput.
- Concurrent Inference Instances - Loads multiple copies of the same model into memory to process several requests in parallel on the same device.
- Inference Performance Monitoring - Tracks GPU utilization, request throughput, and latency to observe overall system health and inference efficiency.
- Server Health Monitoring - Tracks hardware utilization, request latency, and throughput to ensure the health of production AI systems.
- Model Serving - Optimized inference solution for cloud and edge deployments.
- Model Serving & Deployment - Maximizes GPU/CPU utilization for model deployment.
- Model Serving Frameworks - High-performance multi-framework inference server from NVIDIA.
- Model Serving and Deployment - Optimized multi-framework inference server for cloud and edge.
- Model Serving - Optimized multi-framework inference server for cloud and edge.
- Serving Frameworks - Optimized inference solution for cloud and edge environments.
