vllm-project/semantic-router

Features

• Inference Gateways - Accepts OpenAI-style chat and responses API requests and dispatches them to the appropriate backend model.
• Intent Classification Pipelines - Reads intent, domain, and safety profile of requests using purpose-built encoders before selecting a handling model.
• Model Request Routing - Sends each request to the model that best balances quality, cost, latency, and privacy.
• Cost-Aware Model Routers - Routes routine traffic to cheaper models and reserves expensive frontier models for requests that need them.
• AI Request Routing - Routes AI inference requests to the optimal model based on semantic meaning, cost, and safety signals.
• Chat Completion Services - Accepts OpenAI-style chat completion requests and routes them to appropriate backends.
• Inference Routing Protocols - Specifies a content-level classification and semantic routing framework as an IETF protocol.
• Inference Cost Optimizers - Sends routine traffic to cheaper models and reserves expensive frontier reasoning only for requests that need it.
• Real-Time Safety Enforcers - Detects and blocks jailbreak attempts, PII leaks, and hallucinations in real time during request routing.
• Multi-Provider Abstractions - Routes inference requests across local, private, and frontier models through a single normalizing layer.
• GPU Fleet Orchestrators - Manages model selection, fleet sizing, and cost optimization across heterogeneous GPU fleets and providers.
• Model Routing - Matches each query to the best model based on extracted signals for efficient mixture-of-models collaboration.
• Model Routing Layers - Routes requests across local, private, and frontier models through a single layer from edge devices to the cloud.
• Model Selection Tools - Routes simple tasks to smaller models and complex tasks to larger ones, minimizing token usage and expense.
• Cost-Performance Optimizers - Routes simple tasks to cheaper models and complex tasks to larger ones to minimize cost.
• Routing Signal Extractors - Extracts request, safety, follow-up, and preference signals from 16 families to inform routing decisions.
• Semantic Routers - Routes inference requests to the optimal model based on semantic meaning, cost, latency, and safety signals.
• Semantic Signal Fusion Engines - Combines multiple semantic signals with AND/OR logic to select the optimal model and configuration.
• Sensitive Data Redaction - Scans requests and responses for sensitive data and applies configurable policies to protect privacy.
• Workload-Based Model Selectors - Routes each inference request to the model best suited for its task, optimizing for latency, cost, or accuracy.
• Semantic Caching - Caches responses for semantically similar queries using vector-based matching to reduce latency and cost.
• Semantic Query Routing - Routes inference requests to the optimal model based on semantic meaning of the input.
• Global Defaults - Provides typed built-in defaults for router, services, stores, and model catalog settings that can be selectively overridden.
• Research Surfaces - Serves as a research surface that lets teams add new signals, algorithms, and plugins without rewriting the serving path.
• Multi-Function Plugin Chains - Extends request/response processing with plugins for caching, security, data protection, and threat detection.
• Toggleable Plugin Chains - Inspects and modifies requests and responses through an extensible plugin chain with per-decision toggling.
• AI Model Selection Signals - Combines multiple semantic signals with AND/OR logic to select the best model for each request.
• LLM Threat Interceptors - Intercepts and blocks jailbreak, PII, and hallucination risks before they reach a model.
• Signal-Based Routing Policies - Evaluates requests against configurable signals and projection rules to select the best model and route.
• Signal-Based Routing Rules - Evaluates AND/OR decision rules over extracted signals and projections to select the active route and model candidates.
• Input Modality Detectors - Detects text, image, or audio inputs and routes them to capable models.
• AI Inference - Defines routing logic using a DSL that owns model cards, signals, projections, and decisions.
• Adversarial Input Detection - Identifies jailbreak attempts and prompt injections in real time to prevent unauthorized model behavior.
• Jailbreak Detectors - Identifies and blocks attempts to circumvent AI safety measures using a binary classification model.
• Prompt Injection Detectors - Blocks prompt injection and jailbreak attempts by inspecting incoming requests for malicious patterns.
• Multi-Threat Policy Enforcers - Detects PII, jailbreak attempts, and hallucinations while logging all security decisions for audit trails.
• LLM Safety Enforcers - Enforces safety and compliance by blocking jailbreak, PII, and hallucination threats at the routing layer.
• PII Detection and Screening - Identifies personal data in queries to protect user privacy using a dedicated classification model.
• LLM Safety Enforcers - Enforces safety by blocking jailbreak, PII, and hallucination threats at the routing decision layer.
• LLM Request Scanners - Scans incoming requests for jailbreak attempts, PII leaks, and hallucinations before model execution.
• YAML Configuration Files - Edits canonical config.yaml files to define listeners, providers, and routing rules for AI inference.
• Policy-to-Artifact Compilers - Translates YAML policy files into verified decision nodes, Kubernetes artifacts, and protocol gates.
• Orchestration Artifact Compilers - Compiles declarative routing policies into Kubernetes artifacts and protocol-boundary gates for deployment.
• Routing Plugin Systems - Adds new processing logic and signal types to routing behavior through configuration without modifying core code.
• Request-Response Filter Chains - Inspects and modifies requests and responses through an extensible chain of togglable plugins.
• Routing Logic Decouplers - Moves routing logic out of application code into reusable signals, decisions, and configuration.
• Signal Combination Logic - Combines multiple semantic signals with AND/OR logic to select the optimal model and configuration.
• AI Provider Routing - Extends inference APIs to support multi-provider routing for agentic AI workloads.
• Multimodal Response Normalizers - Routes multimodal and image-generation requests to backends and normalizes responses.
• Provider-Agnostic LLM Routing - Coordinates requests across local, private, and frontier models using a single declarative policy layer.
• AI Model - Adds new signals, algorithms, and plugins for AI model routing without rewriting the serving path.
• Fleet Sizing Dashboards - Provides a dashboard for simulating and sizing LLM GPU fleets to meet latency targets.
• Cross-Encoder Rerankers - Applies joint cross-attention scoring to query-candidate pairs for high-precision reranking.
• Hallucination Detection - Analyzes token-level output in real time to flag factual inaccuracies as the model generates text.
• Responses API Translators - Accepts OpenAI Responses API requests and translates them to chat completions for routing.
• Reasoning Need Detectors - Determines whether a query requires reasoning and applies expensive reasoning models only when beneficial.
• Token Budget Routers - Estimates token budgets and dispatches requests to short or long context pools to cut fleet cost.
• Context Length Routers - Routes requests based on context length to improve energy efficiency and reduce fleet cost.
• Request-Response Translation Layers - Converts OpenAI-style chat requests to Anthropic's Messages API format and translates responses back.
• Cross-Encoder Rerankers - Applies joint cross-attention scoring to query-candidate pairs for high-precision reranking.
• RAG Grounding Verifiers - Checks long-document RAG responses for grounding against source contexts up to 32K tokens in real time.
• Domain Classifiers - Routes queries to specialized models based on academic or professional domains using a fine-tuned classifier.
• Spending Controls - Reserves expensive model capabilities for high-value requests and uses caching and routing to reduce waste.
• AI Token Spend Controllers - Reserves premium models and long context for high-value requests using caching and context-aware routing.
• Category-Based Caches - Caches query results by category with per-category similarity thresholds and TTLs.
• Memory-Augmented Model Routers - Uses conversational memory and retrieval to let lightweight models match larger model performance on persistent queries.
• Difficulty-Based Routers - Estimates action difficulty for agent steps and routes to the cheapest model meeting a reliability threshold.
• Semantic Search - Encodes queries and candidates into dense vectors to find semantically similar matches for caching or retrieval.
• Dense Vector Rankers - Encodes queries and candidates into dense vectors for similarity search and relevance scoring.
• Fleet Sizing What-If Simulators - Replays traces and tests planning assumptions through simulation to validate fleet-sizing decisions.
• GPU Fleet Capacity Simulators - Sizes multi-pool LLM GPU fleets against latency targets using discrete-event simulation.
• GPU Fleet Simulators - Simulates homogeneous, heterogeneous, or disaggregated GPU fleets to determine the configuration that meets a given latency target.
• GPU Fleet Cost Comparators - Compares yearly cost across GPU types, routing policies, and threshold settings for fleet optimization.
• ML Policy Conflict Detectors - Identifies when probabilistic ML predicates in routing policies silently co-fire on the same query.
• Security Decision Loggers - Logs all security decisions and applies model-specific PII policies to meet regulatory requirements.
• Token-Level Sensitive Span Detectors - Labels individual tokens to identify PII and safety-sensitive spans requiring localized intervention.
• Web-Based Configuration Dashboards - Walks through model setup, preset selection, and config activation using a browser-based interface.
• LLM Fleet Capacity Planners - Sizes multi-pool LLM GPU fleets against P99 time-to-first-token targets using discrete-event simulation.
• Workload CDF Optimizers - Derives the minimum-cost two-pool LLM fleet directly from the workload cumulative distribution function and latency target.

Historique des stars