kubeflow/kfserving

Features

• Kubernetes Custom Resource Deployments - Defines inference services as Kubernetes custom resources with declarative YAML for model deployment and lifecycle management.
• Kubernetes ML Platforms - Deploys and manages ML models as scalable inference services on Kubernetes with autoscaling.
• OpenAI-Compatible APIs - Serves endpoints matching the OpenAI API format for completions, chat, embeddings, and reranking.
• Graph-Based Inference Routings - Routes inference requests through configurable directed graphs of model steps.
• Generative AI Model Serving - Deploys and serves large language models with optimized backends like vLLM and GPU acceleration.
• LLM Controller Deployments - Installs the LLM Inference Service controller with KV-cache management and multi-node serving for LLMs.
• vLLM Backend Runtimes - Serves large language models for text generation, chat, embeddings, and reranking using vLLM with continuous batching.
• Traffic Management - Routes, rate-limits, and authenticates inference requests across model versions and replicas.
• Inference Pipeline Orchestrators - Orchestrates complex inference workflows by chaining, ensembling, and routing through multiple models.
• Component-Based Inference Routings - Directs traffic between predictor, transformer, and explainer components in inference pipelines.
• Model-Aware Autoscaling - Scales serving instances based on token throughput, queue depth, and GPU utilization, including scale-to-zero.
• Large Language Model Serving - Runs large language models in production with dedicated support for performance and reliability requirements.
• Kubernetes Deployments - Creates Kubernetes deployments for single LLMs, managing Pods, Services, and networking.
• Multi-Source Model Loaders - Downloads model files from cloud storage, HTTP, Git, PVCs, Hugging Face, or OCI images into a local serving directory.
• Graph-Based Inference Routings - Orchestrates complex multi-model inference pipelines through directed graphs of model steps.
• Model-as-a-Service Platforms - Packages trained models into scalable inference services that run on Kubernetes and respond to prediction requests.
• Inference Graph Ensembles - Orchestrates multiple models into inference graphs for chaining and ensemble operations.
• Model Serving & Deployment - Installs the core controller managing InferenceService, ServingRuntime, and InferenceGraph resources for ML model serving.
• vLLM Engines - Runs large language models using a high-performance inference engine optimized for throughput and latency.
• Multi-Framework Model Serving - Runs exported models from TensorFlow, PyTorch, Scikit-learn, XGBoost, and others behind a unified inference endpoint.
• Model Serving Runtimes - Configures execution environments for models as reusable pod templates scoped to namespaces or clusters.
• Pluggable Runtimes - Provides a pluggable runtime abstraction layer that maps model formats to containerized execution environments.
• Reusable Pod Templates - Defines reusable pod templates for serving runtimes, enabling custom engines without modifying controller code.
• Multi-Model Orchestration - Chains multiple models into directed graphs for complex inference workflows.
• Multi-Model Workflow Coordinators - Orchestrates ensembles and pipelines of models for complex multi-step inference.
• Prefill-Decode Disaggregation - Separates prefill and decode phases across different compute pools to optimize cost and performance for LLM serving.
• Model Lifecycle Management - Manages the full lifecycle of AI model serving resources from creation through scaling and cleanup.
• Model Weight Caching - Caches large model weights on local node storage to reduce startup time from minutes to seconds.
• Kubernetes Node Caches - Downloads and stores models on Kubernetes nodes so inference services load from local storage instead of a remote URI.
• Model Storage Backends - Loads models from S3, GCS, Azure, HuggingFace, PVC, and other storage backends.
• Knative-Based Deployments - Runs model servers on Knative Services with scale-to-zero and revision management.
• Remote Model Loading - Fetches model artifacts from S3, GCS, Azure Blob, or Hugging Face Hub for deployment.
• Canary Deployment Controllers - Manages incremental traffic shifts to new model versions for safe validation.
• LLM Deployment Operators - Deploys and manages LLM inference workloads on Kubernetes using a custom resource definition.
• Model Deployment Management - Manages the full lifecycle of model serving with canary rollouts and monitoring.
• Local Weight Caches - Caches large model weights on local nodes to cut startup time from minutes to seconds.
• Scale-to-Zero Pods - Automatically scales model-serving pods down to zero when idle and back up on demand.
• Knative-Based Serverless Deployments - Leverages Knative serving for automatic scale-to-zero, revision management, and traffic splitting.
• Inference Workload Autoscalings - Scales serving instances automatically based on token throughput, queue depth, or GPU utilization, including scale-to-zero.
• Serving Resource Autoscalings - Scales model replicas up or down based on request volume, including scale-to-zero for cost savings.
• Inference Service Rollouts - Routes configurable traffic percentages to new model revisions with automatic rollback.
• Model Inference Deployments - Runs inference workloads on a serverless platform with automatic scaling based on request volume.
• Storage Backend Integrations - Loads models from external storage sources such as HuggingFace, S3, or PVC volumes for deployment.
• Revision Traffic Splits - Splits inference traffic across model revisions for canary testing and gradual rollouts.
• Serving Engine Swappers - Packages custom serving logic as pluggable runtimes to support proprietary or specialized model engines.
• Multi-Protocol Verifiers - Verifies API keys, OAuth2 tokens, or Kubernetes service accounts before allowing model access.
• AI Service Access Policies - Validates API keys and enforces fine-grained access control policies for backend AI services.
• Model Pipeline Orchestration - Coordinates inference requests across chains of models with conditional logic and branching.
• Model-Aware Request Routings - Distributes inference requests across model versions and replicas based on load, caching, and QoS requirements.
• AI Gateways - Directs LLM inference traffic through a high-performance gateway specialized for generative AI workloads.
• Parallelism Strategy Configurations - Distributes model inference across multiple nodes using tensor, data, or expert parallelism for scaling large models.
• Generation Parameter Configurations - Tunes model behavior with temperature, top-p, and system instructions during inference.
• Hugging Face Text Generation Runtimes - Loads and runs Hugging Face Transformers models for text generation tasks within a Kubernetes inference platform.
• Multi-Node Inference Scaling - Distributes model inference across multiple nodes and GPUs for higher throughput.
• Deployments - Loads a trained XGBoost model into a Kubernetes-native inference service and exposes it for real-time predictions.
• Outlier Detection - Flags individual input instances that deviate from the training distribution to identify anomalous requests.
• In-Memory Model Caching - Caches frequently used models in memory to reduce load times and improve response latency.
• Activation and KV Cache Offloaders - Moves key-value cache data from GPU to CPU memory to free GPU resources and serve larger models.
• KV-Cache-Aware Request Routing - Routes requests to optimal model replicas using KV-cache aware scheduling.
• LLM Response Streaming - Streams output token-by-token as generated using Server-Sent Events for low-latency LLM experiences.
• Scikit-Learn Inference Servers - Loads and runs Scikit-Learn models as scalable inference services on Kubernetes.
• Parallelism Configuration - Sets tensor, pipeline, and data parallelism parameters to distribute inference work across multiple GPUs or nodes.
• Ensemble Prediction Combinations - Combines predictions from multiple models using majority vote or averaging.
• Model Response Synthesis - Runs multiple inference steps concurrently and merges outputs into a single response.
• ONNX Model Runtimes - Loads and runs ONNX models for cross-framework inference on Kubernetes.
• Model Storage Management - Configures how and where model artifacts are stored and accessed by the serving infrastructure.
• Model Versioning - Handles storage, retrieval, and lineage tracking for multiple versions of a trained model.
• LoRA Adapter Loaders - Attaches task-specific LoRA adapters to a base model for efficient multi-tenant serving without full model duplication.
• Sequential Step Orchestrators - Executes inference steps in order, passing outputs as inputs to subsequent steps.
• TensorFlow Serving Systems - Loads and runs TensorFlow SavedModel or frozen graph formats for inference on Kubernetes.
• Token Usage Monitoring - Automatically counts input, output, and total tokens per request for cost monitoring and rate limiting.
• Cluster Node Caches - Downloads and stores LLM weights on cluster nodes to reduce download times and storage redundancy.
• Pre-Downloaded Node Caches - Pre-downloads models from persistent storage onto a Kubernetes node's local disk so replicas start faster.
• Model Cache Node Groups - Creates groups of Kubernetes nodes with local persistent volumes where models are cached and managed.
• Inference Batching - Accumulates multiple prediction requests and processes them together to increase throughput.
• MLflow Deployments - Deploys MLflow-packaged models as scalable inference services on Kubernetes without manual containerization.
• Named Serving Runtime Selections - Specifies named serving runtimes in InferenceService predictors for model deployment.
• LLM Replica Autoscaling - Scales inference replicas based on token throughput, queue depth, and GPU utilization metrics.
• Standard Kubernetes Deployments - Runs model servers on standard Kubernetes Deployments with HPA scaling.
• Inference Container Deployments - Packages custom model-serving logic into a container image and runs it as a scalable inference service on Kubernetes.
• Model Rollout Strategies - Rolls out new model versions gradually using canary deployments, inference pipelines, and ensembles.
• Data Drift Detectors - Detects statistical shifts between incoming request distributions and training data to identify model degradation.
• Token Consumption Limiters - Restricts token consumption per user and model using configurable rate limits to control costs and prevent overuse.
• Ingress Gateways - Creates Kubernetes Ingress or Gateway API HTTPRoutes to expose inference services externally.
• Hub Deployments - Deploys models directly from the Hugging Face Hub with minimal configuration.
• Transformer Model Runtimes - Deploys Hugging Face transformer models from the Hub or custom sources using a specialized predictor runtime.
• Local Cache Deployments - Deploys inference services that load models from a local cache on node storage to reduce startup time.
• Inference Pipeline Hooks - Applies built-in data transformation pipelines and feature engineering as pre- or post-processing steps.
• Token-Based Rate Limiters - Routes generative inference requests to different models and enforces token-based rate limits through a unified AI gateway.
• Workload-Type Tuners - Adjusts GPU, memory, and autoscaling settings based on whether the workload is generative or predictive.
• Standardized Inference Protocols - Implements a standardized V2 inference protocol with REST and gRPC endpoints for model communication.
• gRPC Implementations - Exchanges inference data over gRPC for low-latency, high-throughput model communication.
• Dual-Protocol Prediction Endpoints - Exposes model inference endpoints over both REST and gRPC for broad client compatibility.
• Conditional Routing - Evaluates conditions on input and executes only the first matching inference step.
• Weighted Traffic Splitting - Routes inference requests across model versions using configurable probability weights for gradual rollouts.
• Conditional Route Matching - Evaluates routing conditions on each request and directs it to the first matching inference step.
• Model Format Runtime Selections - Automatically selects serving runtimes for model formats using version and priority matching.
• End-to-End Message Tracing - Propagates trace context across inference pipeline boundaries for end-to-end request diagnostics.
• Monitoring and Observability - Detects data drift, flags outliers, tracks token usage, and collects metrics for deployed models.
• Model Performance Monitoring - Monitors deployed model behavior by logging payloads and detecting outliers, adversarial inputs, and drift.
• OpenTelemetry-Integrated Monitors - Collects traces, metrics, and logs from inference services using OpenTelemetry for performance monitoring.
• Prometheus Metrics Exporters - Gathers time-series metrics from inference services in Prometheus format for monitoring and alerting.
• Inference Payload Logs - Captures every prediction request and response for auditing, debugging, and compliance purposes.
• Inference Data Transformers - Applies built-in data transformation pipelines and feature engineering as pre- or post-processing steps.
• Model Serving - Kubernetes-native framework for serving models on various frameworks.
• Model Serving Frameworks - Kubernetes-native serverless inference for machine learning models.

Historique des stars