Text Embedding Generation Libraries
For a library for generating text embeddings, the strongest matches are jina-ai/clip-as-service (CLIP-as-service is a deployable service that generates text and), huggingface/text-embeddings-inference (Text Embeddings Inference is a high-performance, self-hostable API server) and huggingface/transformers (Hugging Face Transformers gives you access to thousands of). google-research/bert and superlinked/superlinked round out the shortlist. Each is ranked by relevance to your query, popularity and recent activity.
Open-source frameworks and tools for converting natural language text into high-dimensional vector representations for search.
Text Embedding Generation Libraries
Find the best repos with AI.We'll search the best matching repositories with AI.
- jina-ai/clip-as-service
jina-ai/clip-as-service
12,829View on GitHubClip-as-service is a deployable framework for generating multi-modal embeddings and executing neural searches. It provides a vector embedding server and a CLIP embedding API to convert images and text into shared vector representations via network interfaces. The system functions as a multi-modal ranking system and neural search engine, enabling the retrieval of images through text queries or the identification of matching text descriptions for images. It also includes a visual reasoning service used to analyze images and verify object presence, counts, and colors by comparing visual data aga
CLIP-as-service is a deployable service that generates text (and image) embeddings using the CLIP transformer model via a REST API, supporting batching and similarity search, directly matching the need for a self-hostable text embedding library with many of the requested features.
PythonEmbedding ServersInference Batching - huggingface/text-embeddings-inference
huggingface/text-embeddings-inference
4,871View on GitHubText Embeddings Inference is a high-performance inference server designed to host text embedding and sequence classification models as scalable API endpoints. It provides a vector embedding API to convert text into dense representations and a cross-encoder reranking server for scoring the relevance of document sequences against a query. The project features a GPU-accelerated inference engine that utilizes dynamic batching and specialized kernels to maximize throughput. It offers a high-performance binary interface via gRPC as an alternative to standard HTTP to reduce network latency and seria
Text Embeddings Inference is a high-performance, self-hostable API server that generates text embeddings from Transformer models with GPU acceleration, batch processing, and gRPC support, directly matching the need for a tool to produce embeddings for semantic search and vector retrieval.
RustEmbedding Servers - huggingface/transformers
huggingface/transformers
161,630View on GitHubTransformers is a comprehensive library for machine learning that provides a unified interface for training, fine-tuning, and deploying transformer-based models. It supports a wide range of tasks, including text classification, language modeling, question answering, and sequence-to-sequence translation, while offering specialized architectures for both text and vision processing. The framework includes tools for managing the entire model lifecycle, from data preprocessing and tokenization to distributed training and inference. The library features extensive support for model optimization and
Hugging Face Transformers gives you access to thousands of pre-trained transformer models for generating text embeddings, supports batch processing and GPU acceleration out of the box, and can be self-hosted with a Python SDK — it's the flagship library for this exact purpose.
PythonBatched Inference Mechanisms - google-research/bert
google-research/bert
39,869View on GitHubThis project is a transformer-based language model and natural language processing toolkit designed to generate deep contextual representations of text. By utilizing a transformer-based encoder architecture, the system processes input sequences through stacked self-attention layers to capture the semantic meaning of tokens based on their surrounding sentence structure. The model distinguishes itself through bidirectional contextual processing, which analyzes text in both directions simultaneously, and masked language modeling, which trains the system by predicting hidden tokens within a seque
BERT is a transformer-based model that generates contextual text embeddings, fitting the core need of this search, but it lacks built-in REST/gRPC APIs or direct similarity search integration, so it covers the essential embedding generation while requiring additional tooling for the full feature set.
PythonTransformer Encoders - superlinked/superlinked
superlinked/superlinked
Superlinked is a development framework designed for building semantic search and retrieval pipelines. It functions as a machine learning data pipeline and semantic retrieval engine, providing the tools necessary to unify data schema definition, embedding generation, and vector database integration within a single application. The framework distinguishes itself by acting as a vector database orchestrator that manages the lifecycle of machine learning models alongside complex search logic. It enables developers to construct structured data models that map raw content and metadata into unified r
Superlinked is a development framework that generates text embeddings as part of its semantic search and retrieval pipeline, covering multiple models and vector database integration—it fits the text embedding generation library category with a broader orchestrator approach.
Jupyter NotebookVector Database IntegrationsVector Database Integrations - flagopen/flagembedding
FlagOpen/FlagEmbedding
11,833View on GitHubFlagEmbedding is a comprehensive toolkit designed for training, benchmarking, and deploying embedding models, retrieval systems, and augmented generation pipelines. It provides the necessary infrastructure to transform text into high-dimensional vector representations and organize them into searchable structures for semantic search applications. The framework distinguishes itself through specialized capabilities for fine-tuning pre-trained embedding and reranking models on domain-specific datasets. By allowing users to adapt models to unique vocabularies and specialized retrieval tasks, it en
FlagEmbedding is a comprehensive toolkit for generating text embeddings, supporting multiple transformer-based models, GPU acceleration, fine-tuning, and deployment with similarity search integration, which fully matches the visitor's need for a self-hostable embedding generation library.
PythonEmbedding GeneratorsEmbedding Model Fine-TuningRetrieval Augmented Generation - ggerganov/llama.cpp
ggerganov/llama.cpp
116,912View on GitHubllama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across various hardware architectures. It provides the core components for local model execution, including a dedicated model quantizer for compressing weights into the GGUF format and a system for generating text embeddings for semantic search. The project distinguishes itself through specialized memory and execution optimizations, such as block-wise weight quantization to reduce memory footprints and memory-mapped model loading. It supports structured text generation by using formal
llama.cpp is a high-performance local inference engine that generates text embeddings from LLMs using transformer architecture, with GPU acceleration, batch processing, and an OpenAI-compatible API — a comprehensive and self-hostable tool for your embedding generation needs.
C++Local Inference EnginesC++ Inference RuntimesCross-Platform Inference Frameworks - hanxiao/bert-as-service
hanxiao/bert-as-service
12,831View on GitHubThis project is a high-performance BERT embedding service and inference server designed to map text sequences into fixed-length numerical vectors. It functions as a machine learning microservice and distributed model server that decouples request handling from heavy computation. The system utilizes a ZeroMQ messaging infrastructure to provide low-latency communication between distributed clients and the inference server. It incorporates server-side batch processing and GPU workload scaling to maximize hardware utilization and manage high request volumes. The platform supports semantic search
bert-as-service is a high-performance BERT embedding service that generates fixed-length vectors from text using a transformer model, with batch processing and GPU support—it delivers on the core embedding generation need, though it focuses on a single model architecture rather than supporting multiple embedding models.
PythonInference Batching - explosion/spacy
explosion/spaCy
33,688View on GitHubspaCy is a Python natural language processing framework designed for industrial-scale text processing. It converts raw text into structured data for machine learning pipelines through a combination of statistical language model trainers, transformer-based text processors, and syntactic dependency parsers. The project enables the integration of pretrained transformer architectures to perform complex linguistic analysis and multi-task learning. It also provides a specialized system for neural named entity recognition to identify and categorize key entities within text. The framework covers a b
spaCy is a full NLP library that generates text embeddings via transformer models, supports batch processing and GPU acceleration, and includes similarity comparison, but it does not provide a built-in REST or gRPC API, which the search specifically requested.
PythonDeep Learning NLP FrameworksIndustrial NLP PipelinesLanguage Model Trainers - ukplab/sentence-transformers
UKPLab/sentence-transformers
18,822View on GitHubThis project is a framework for training and deploying transformer-based models that map text, images, audio, and video into dense or sparse vector representations. It functions as a multimodal embedding library and semantic search engine used to retrieve relevant documents by calculating vector similarity between meanings. The framework provides specialized tools for both cross-encoder reranking, which calculates precise similarity scores to refine search results, and vector quantization to compress embedding vectors for reduced memory usage and increased retrieval speed. The project covers
Sentence Transformers is the premier Python library for generating text embeddings with transformer models, supporting multiple model architectures, batch processing, GPU acceleration, and direct integration with similarity search—fully hitting your core requirement, though it lacks a built-in REST/gRPC API.
PythonEmbedding GeneratorsSentence EmbeddingsContrastive Learning Models - facebookresearch/infersent
facebookresearch/InferSent
2,279View on GitHubInferSent sentence embeddings
InferSent is a library that generates sentence embeddings from text, which matches the core need, but it uses an LSTM architecture rather than transformers and does not include a REST API or built-in similarity search, making it a narrower option for your search.
Jupyter NotebookGenerative ModelsNatural Language Processing - xlang-ai/instructor-embedding
xlang-ai/instructor-embedding
2,024View on GitHubInstructor-embedding is a natural language processing framework designed to transform unstructured text into high-dimensional numerical vectors. By utilizing a transformer-based encoder architecture, the system facilitates semantic retrieval, data classification, and similarity analysis across large datasets. The framework distinguishes itself through instruction-conditioned vector projection, which incorporates natural language instructions directly into the embedding process to improve performance for specific tasks without requiring additional training. It functions as a contrastive learni
Instructor Embedding is a Python library that generates instruction-finetuned text embeddings, directly matching your need for a transformer-based text embedding generator, though it lacks a built-in REST/gRPC API and may require additional setup for server deployment.
PythonTransformer Encoders
