shibing624/text2vec
Text2vec
text2vec 是一个文本向量化工具包和语义相似度框架，用于将单词和句子转换为数值向量。它提供了用于生成嵌入、计算语义接近度以及实现词法和语义搜索的集成工具集。
该项目包括一个模型微调流水线，用于使用监督或无监督数据集优化嵌入和匹配模型。它还通过提供文本嵌入 API 进一步脱颖而出，该 API 允许通过 gRPC 或 HTTP 协议将向量化模型部署为网络服务。
该框架涵盖了广泛的功能，包括语义分析、神经文本编码和向量搜索检索。它支持密集向量嵌入和诸如 BM25 之类的词项重叠算法，以确定相似度并识别相关词项。
额外的性能优化包括模型蒸馏、基于 PCA 的降维以及针对大规模数据集的多 GPU 分布式推理。
Features
- Semantic Similarity Calculation - Calculates conceptual similarity between text segments using cosine similarity and embedding vectors.
- Text Embeddings - Converts text into dense numerical vectors using transformer-based models to represent semantic meaning.
- Text Vectorizations - Transforms words and sentences into numerical vectors to represent semantic meaning for ML tasks.
- Text Vectorization Tools - Provides a toolkit for converting text into vectors using models like Word2Vec and Sentence-BERT.
- Hybrid Semantic Search Tools - Provides a system for retrieving documents using both BM25 lexical overlap and dense vector embeddings.
- Vector Similarity Search - Implements vector similarity search using cosine distance to measure semantic closeness between text embeddings.
- Model Fine-Tuning - Provides a dedicated pipeline for optimizing embedding and matching models using supervised or unsupervised datasets.
- Transformer Text Encoders - Uses transformer-based encoders to generate dense numerical vectors that represent the conceptual meaning of text.
- Semantic Search Engines - Implements a semantic search engine that retrieves relevant documents based on conceptual query meaning.
- Embedding Model Fine-Tuning - Implements processes for fine-tuning embedding models to optimize vector representations for specific domains.
- Model Distillation - Implements model distillation to create smaller, faster versions of embedding models without significant accuracy loss.
- Cross-Encoders - Provides cross-encoders that process text pairs simultaneously for high-precision semantic similarity scoring.
- Model-as-a-Service Platforms - Packages vectorization models into scalable inference services as a Model-as-a-Service implementation.
- Contrastive Fine-Tuning - Provides a pipeline for supervised contrastive fine-tuning to improve domain-specific embedding accuracy.
- Semantic Vector Search - Retrieves the most relevant documents from a dataset based on the mathematical distance between query and document embeddings.
- Inference Service Hosting - Wraps vectorization functions into high-performance API services for cloud-native deployment.
- Model Inference APIs - Exposes text vectorization models as high-performance gRPC and HTTP network services.
- BM25+ Scoring - Supports lexical-overlap scoring using the BM25 algorithm for non-semantic document retrieval.
