huggingface/text-embeddings-inference

Features

• High Throughput Inference - Optimizes hardware utilization with dynamic batching and GPU acceleration to process large volumes of text requests.
• Embedding Servers - Acts as a high-performance network-accessible server for generating high-dimensional vector representations.
• Dynamic Batching Engines - Groups multiple individual requests into a single GPU operation to maximize hardware throughput and reduce compute overhead.
• Document Rerankers - Provides a dedicated reranking server to score the relevance of documents relative to a query across multiple languages.
• GPU-Accelerated Inference - Utilizes a compute-optimized runtime with dynamic batching and specialized kernels to maximize GPU throughput.
• Throughput Optimizers - Implements high-throughput inference using dynamic batching and optimized transformer kernels to maximize GPU utilization.
• Cross-Encoder Rerankers - Implements a high-precision reranking server using joint cross-attention scoring for query-document pairs.
• Retrieval Re-ranking - Uses cross-encoder models to score the relevance of candidate texts against a query to improve search accuracy.
• Specialized Mathematical Kernels - Uses specialized low-level mathematical implementations to accelerate the core attention and linear layers of embedding models.
• Text Classification - Serves models that categorize and label text inputs for predicting sentiment or emotions.
• Text Embedding Generators - Transforms text inputs into dense vector representations using instruction-based queries.
• Text Embeddings - Provides a scalable server for generating dense vector representations from text for AI applications.
• Vector Embeddings - Provides an API that converts text into dense vector representations for semantic search.
• Cross-Encoder Scoring - Uses cross-encoder models to calculate similarity scores for query-passage pairs to refine search results.
• ML Model Hosting - Supports loading and hosting model weights from remote repositories or local directories.
• AMD Hardware Acceleration - Enables embedding and classification models to run on AMD hardware using the ROCm compute platform.
• Apple Hardware Acceleration - Provides hardware acceleration for embedding and classification models locally on macOS using Apple Silicon.
• RAG Pipeline Integrations - Generates high-performance embeddings and similarity scores to power retrieval-augmented generation pipelines.
• Hardware Dispatchers - Directs model execution to optimized compute paths for NVIDIA GPUs, AMD GPUs, or Apple Metal based on the host.
• Long Context Processing - Supports the processing of large input sequences consisting of several thousand tokens for extended document context.
• Memory-Mapped Weight Loaders - Maps model weights directly from disk into memory to enable rapid startup and efficient resource sharing across processes.
• Similarity Query Engines - Uses cross-encoder models to rank the semantic closeness of documents relative to a query.
• Batch Input Processing - Handles multiple text inputs in a single request to increase total inference throughput.
• Model Endpoint Deployment - Creates hosted environments with specific hardware accelerators and runtime configurations for model inference.
• Air-Gapped Deployments - Supports running models in isolated networks by mounting pre-downloaded weights as volumes.
• gRPC Interfaces - Implements a high-performance binary interface via gRPC to reduce serialization latency and network overhead.
• Text Embedding gRPC Services - Provides a high-performance gRPC binary interface to reduce network latency compared to standard HTTP.
• Concurrent Request Limits - Limits the number of simultaneous requests per instance to prevent timeouts and manage back pressure.
• Inference and Serving - Optimized inference for text embeddings in Rust.
• Model Serving Engines - Dedicated inference server for text-embedding models.

Star history