tensorflow/serving
Serving
TensorFlow Serving is a high-performance machine learning inference server designed to deploy TensorFlow models to production environments. It functions as a complete serving system that executes predictions on input data through a graph executor, providing network endpoints that eliminate the need for a separate runtime environment for client applications.
The system is distinguished by its model version manager, which organizes and selects specific model versions within a directory hierarchy. It uses a filesystem watcher to detect new model versions and trigger automatic updates without interrupting live traffic.
Connectivity is provided through dual gRPC and REST API gateways that map input and output tensors to named serving signatures. The platform includes capabilities for large model export to bypass filesystem size limits, as well as tools for model metadata inspection and inference testing using sample inputs.
Features
- Model Inference and Serving - Provides a high-performance platform for deploying, optimizing, and serving machine learning models for production use.
- Model Serving & Deployment - Provides a high-performance system for deploying trained models to production environments for scalable inference.
- Model Inference Servers - Functions as a dedicated server application that hosts machine learning models to provide scalable network-accessible inference services.
- Model Inference - Generates predictions from input data and processes inference results from loaded models.
- Model Versioning - Organizes different model versions in a directory hierarchy to select specific versions during inference requests.
- Model Versioning Systems - Organizes and switches between different versions of a machine learning model without interrupting live traffic.
- TensorFlow Graph Execution - Implements a runtime for processing TensorFlow computation graphs and mapping input tensors to named output functions.
- TensorFlow Model Inference - Executes predictions on trained TensorFlow models using gRPC or REST APIs for client applications.
- TensorFlow Serving Systems - Provides a high-performance environment for deploying TensorFlow machine learning models to production via gRPC and REST endpoints.
- Inference Servers - TensorFlow Serving exposes trained models through REST endpoints to process inference requests and return specific predictions.
- Model Inference Deployment - Deploys trained models to production environments to provide high-performance inference via network endpoints.
- C++ Inference Runtimes - Provides a high-performance C++ engine optimized for loading and executing machine learning models to minimize inference latency.
- Function Signature Definitions - Maps input and output tensors to named functions so external tools can identify how to interact with the model.
- Inference Correctness Testing - Enables the execution of graph computations with sample inputs to verify model correctness before production deployment.
- Model Metadata Inspection - Provides a command-line interface to examine computation tags, signature definitions, and tensor shapes of deployed models.
- Model Inference APIs - Exposes HTTP endpoints specifically designed to serve real-time predictions from loaded machine learning models.
- Model Serving Endpoints - Exposes trained models via gRPC and HTTP endpoints to provide high-performance inference for client applications.
- REST-to-gRPC Gateways - Implements gateways that translate RESTful HTTP API calls into internal gRPC requests for the model server.
- Model Version Watchers - Automatically detects new model versions in the filesystem to trigger updates without interrupting live traffic.
- Concurrent Inference Instances - Coordinates concurrent inference requests across multiple loaded models and different version instances.
- Model Serving & Deployment - Serves TensorFlow models with high-performance gRPC.
- Serving Frameworks - High-performance serving system for TensorFlow models.
