SeldonIO/seldon-core

Features

• Inference Pipeline Orchestrators - Orchestrates complex sequences of models and data transformers into multi-stage inference workflows.
• Model Inference and Serving - Functions as a comprehensive platform for deploying and serving machine learning models for production use.
• Standardized Inference Protocols - Implements standardized REST and gRPC protocols for consistent request and response handling across different model runtimes.
• Automatic Inference Workload Scaling - Adjusts the number of inference serving replicas based on demand, including scaling to zero.
• Graph-Based Inference Routings - Orchestrates complex inference workflows through directed graphs that chain models and data transformers.
• MLOps Inference Frameworks - Provides a comprehensive framework for managing the full production lifecycle of inference models.
• Model Deployment Runtimes - Deploys specific server environments tailored for model execution and integrated resource monitoring.
• Graph-Based Inference - Executes AI models by processing a computational graph of connected services and tensors.
• Tensor Alignment - Renames and aligns output tensors from one model to match the required inputs of the next pipeline step.
• Multi-Model Servers - Hosts multiple model artifacts on a single server instance to maximize hardware efficiency and reduce overhead.
• Model Serving & Deployment - Packages and serves machine learning models as scalable microservices accessible via standardized APIs.
• Kubernetes Custom Resource Deployments - Uses Kubernetes custom resources to declaratively define the lifecycle and state of model deployments and pipelines.
• Model Serving Infrastructure - Provides infrastructure to host and scale multiple models on shared hardware using overcommit strategies.
• Pre- and Post-Inference Transformations - Provides pluggable pipelines to transform input data and model outputs for feature engineering and preparation.
• Inference Request APIs - Exposes network endpoints that allow users to send data and retrieve predictions from deployed models.
• Input Distribution Drift - Analyzes input data distributions over time to identify statistical shifts from the original training data.
• Model Drift and Outlier Detection - Monitors incoming data streams to identify statistical drift and anomalous outliers using detector components.
• Model Experimentation - Splits traffic between different model versions to perform A/B testing and shadow deployments.
• AI Infrastructure Deployments - Automates the setup of operators, runtimes, and servers on clusters to provide the foundation for model management.
• Operator-Based Lifecycles - Automates the installation and scaling of model servers using the Kubernetes Operator pattern.
• Model Inference Deployment - Loads models into Kubernetes clusters to process inference requests via a service mesh.
• Kubernetes ML Platforms - Functions as a dedicated platform for deploying and managing machine learning models as microservices on Kubernetes.
• MLOps Infrastructure Management - Manages the full lifecycle, autoscaling, and hardware resource allocation for large numbers of deployed models.
• Model Deployment Management - Provides administrative interfaces to control the lifecycle, scaling, and configuration of deployed AI systems.
• Model Memory Overcommit - Maximizes hardware usage by loading more models than physical memory allows through dynamic resource management.
• Asynchronous Request Processing - Streams input data through a pipeline via a message-bus and delivers results via an output topic.
• Model Pipeline Orchestration - Chains multiple AI models in a directed graph where outputs flow as inputs to subsequent models.
• Inference Protocol Standards - Provides a uniform API standard for inference to ensure interoperability between various model runtimes and clients.
• Model Health Monitors - Tracks real-time operational metrics like latency and error rates to ensure stability of deployed models.
• Server Managers - Deploys and scales sets of server replicas to manage model request volume.
• Dataset Batch Inference - Provides capabilities for executing large-scale inference jobs that process entire datasets in bulk.
• Inference Configurations - Defines custom installation settings and parameters for controlling model inference execution environments.
• Inference Traffic Routing - Directs traffic to appropriate model servers using a high-availability ingress and reverse proxy system.
• Custom Metric Definitions - Allows the definition of user-specified scoring mechanisms tailored to specific model behaviors and business cases.
• Resource-Based Scheduling - Implements scheduling logic to match model resource requirements against available server capabilities.
• Inference Optimizations - Implements techniques to reduce latency and increase throughput during the model inference phase.
• Model Explainability - Provides insights into how models arrive at specific outputs to make production decisions interpretable.
• Environment Dependency Management - Defines custom software dependencies to ensure servers meet the specific environment requirements of hosted models.
• Observability And Monitoring - Includes tools for detecting data drift, monitoring health, and generating prediction explainability reports.
• Model Version Experimentation - Implements routing logic for A/B tests and Canary deployments to compare model versions in production.
• Synchronous Request-Response - Provides a gateway API to interact with the inference pipeline in a blocking request-response manner.
• Real-Time Data Streaming - Integrates real-time data streaming to ensure models utilize current information for predictions.
• Real-time Feature Pipeline Orchestrators - Orchestrates real-time feature pipelines to connect models with streaming data sources for live monitoring.
• Service Mesh - Exposes models through a single endpoint compatible with various service mesh routing implementations.
• Model Deployments - Manages the deployment and scaling of machine learning and large language models across hybrid and multi-cloud environments.
• Model Runtime Configurations - Defines infrastructure and dependency requirements to match models with compatible execution runtimes.
• Inference Pipeline Scaling - Improves data processing and resource scaling across model sequences to resolve performance bottlenecks.
• Model Memory Overcommitment - Optimizes hardware utilization by loading more models than available RAM through dynamic memory overcommit strategies.
• Resource-Metric-Based Auto Scaling - Adjusts the number of model replicas based on resource utilization and lag to optimize hardware usage.
• Traffic Shadowing - Duplicates live requests to a shadow model to evaluate performance without affecting the user.
• Weighted Traffic Splitting - Distributes incoming traffic across model versions using percentage-based weights for A/B testing and canary releases.
• Message Bus Architectures - Decouples inference input and output by streaming requests through a distributed message-bus pipeline.
• Data Pipeline Monitoring - Tracks metrics and logs for data flows to detect anomalies or inefficiencies in production pipelines.
• Model Performance Monitoring - Collects inference statistics and observability data to inform performance monitoring and autoscaling.
• Inference Payload Logs - Captures the full input and output payloads of every pipeline step for auditing and debugging.
• AI and Machine Learning - MLOps framework for deploying and managing production models.
• Machine Learning Operations - Open source platform for deploying and monitoring machine learning models in production.
• Machine Learning Platforms - Framework for deploying and scaling production machine learning models.
• MLOps Platforms - Packages and manages production machine learning models.
• Model Serving & Deployment - Deploys ML models on Kubernetes.
• Model Serving Frameworks - Framework for deploying and managing ML models on Kubernetes.
• GitOps And Continuous Delivery - MLOps framework to package, deploy, and monitor machine learning models.
• Developer Tools - Deploys and manages machine learning models on Kubernetes clusters.

Star history