huichen/wukong
Wukong
Wukong is a distributed full-text search engine designed for indexing and retrieving text documents. It functions as a customizable search backend that employs a BM25 relevance ranker to order search results based on term frequency and inverse document frequency.
The system includes a specialized Chinese text segmenter to break continuous character strings into meaningful words for accurate indexing and retrieval. To handle large datasets and high request volumes, it utilizes a distributed search index that employs hash-based sharding to split documents across multiple nodes.
The engine provides comprehensive information retrieval capabilities, including position-aware token indexing for proximity filtering and multi-criteria result ranking. It supports real-time index management through batched updates and ensures data availability across restarts via index data persistence.
Features
- Full-Text Inverted Indexes - Uses a full-text inverted index to map tokens to document identifiers for fast retrieval.
- Full Text Indexing - Implements a full-text indexing system to enable fast and comprehensive information retrieval from text documents.
- BM25 Search Indices - Employs a BM25-based ranking system to order search results by term frequency and inverse document frequency.
- Chinese Search Engines - Provides a search capability specifically optimized for Chinese linguistic requirements through word segmentation.
- Cross-Shard Result Aggregation - Collects and sorts partial results from multiple shards to produce a consistent global result set.
- Distributed Search Engines - Scales search operations across multiple nodes using a sharded architecture to handle massive datasets.
- Full Text Search - Provides full-text search capabilities to index and retrieve documents using keyword matching and relevance ranking.
- Full-Text Search Engines - Implements a distributed full-text search engine for indexing and retrieving text documents.
- Distributed Data Indexing - Utilizes a sharded distributed index to split documents across nodes, reducing latency and increasing scale.
- Index Sharding - Provides a distributed search index that employs sharding to handle large datasets and high request volumes.
- Hash-Based Index Partitioning - Implements hash-based sharding to distribute the search index across multiple nodes for scalability.
- Distributed Search Engines - Scales search tasks across multiple nodes to handle high request volumes and large-scale data clusters.
- Chinese Language Segmenters - Supports accurate Chinese document discovery through specialized word segmentation and information retrieval.
- Search Ranking Algorithms - Orders search results using BM25 and user-defined scoring rules to prioritize the most relevant content.
- BM25+ Scoring - Implements the BM25 algorithm for calculating document relevance based on term frequency and inverse document frequency.
- Real-time Indexing - Updates searchable text indices in real time using batched operations to keep results current.
- Proximity and Relevance Executions - Integrates BM25 ranking and keyword proximity calculations to execute advanced search queries.
- Index Persistence - Provides durable storage for search indices to ensure data availability across application restarts.
- Token Offset Mapping - Maps tokens to their exact offsets within documents to support proximity filtering and phrase searches.
- Dynamic Index Updating - Supports real-time index management through batched updates to keep search results current.
- Proximity Filtering - Provides position-aware token indexing to enable proximity filtering based on the physical distance between keywords.
- Search Backends - Functions as a customizable search backend supporting token proximity filtering and user-defined scoring rules.
- Multi-Criteria Ranking - Supports multi-criteria result ranking that combines relevance, keyword proximity, and metadata fields.
- Relevance Scoring - Calculates document relevance using user-defined scoring logic and custom data fields.
- Search and Recommendation - Full-text search engine.
- Search and Recommendation - Full-text search engine.
