go-ego/riotArchived
Riot
Riot es un motor de búsqueda distribuido y servidor de indexación basado en Go diseñado para indexación y recuperación de texto completo. Funciona como un sistema de recuperación que ordena documentos por relevancia utilizando algoritmos de ranking BM25, frecuencia de términos y frecuencia inversa de documento (TF-IDF).
El motor proporciona soporte especializado para el idioma chino, con segmentación de texto concurrente y mapeo fonético Pinyin para hacer coincidir la entrada romanizada con los caracteres. Utiliza una arquitectura distribuida que emplea fragmentación (sharding) de índice basada en hash para equilibrar la carga de datos y el rendimiento a través de múltiples nodos de servidor.
El sistema cubre una amplia gama de capacidades de búsqueda, incluyendo ejecución de consultas con lógica booleana, filtrado de proximidad y gestión del ciclo de vida del índice en tiempo real. Mantiene un índice de búsqueda rápido en memoria mientras utiliza almacenamiento respaldado por disco para la persistencia y durabilidad de los datos. Se incluyen herramientas de monitoreo para rastrear la utilización de memoria, disco y CPU en todo el entorno distribuido.
Features
- Full-Text Search Indexes - Provides full-text indexing and retrieval of large textual datasets using keyword-based searches.
- Distributed Search Engines - Functions as a scalable, high-performance distributed search engine platform for indexing and retrieving massive volumes of text data.
- Pinyin Character Mappings - Provides phonetic Pinyin mapping to allow users to match romanized input with Chinese characters.
- BM25 Search Indices - Functions as a retrieval system that sorts documents by relevance using the BM25 algorithm.
- Search Query Composition - Implements a query language that combines search terms with AND, OR, and AND_NOT operators to refine results.
- Chinese Search Engines - Provides a specialized search engine with tokenization and Pinyin support for Chinese text retrieval.
- Cross-Shard Result Aggregation - Implements cross-shard result aggregation to combine and re-rank search results from multiple distributed nodes.
- Distributed Query Engines - Implements a distributed query engine that aggregates search results from multiple independent storage nodes into a unified output.
- Full Text Search - Provides keyword-based search capabilities with configurable ranking and offset options for document retrieval.
- Information Retrieval - Implements information retrieval ranking using BM25 and proximity scoring to surface relevant documents.
- Search & Indexing Engines - Builds distributed searchable indexes of large data volumes across multiple nodes with real-time updates.
- Full Text Indexing - Implements full-text indexing by adding content to a searchable index and assigning unique document identifiers.
- Index Sharding - Distributes searchable indexes across multiple servers using hash-based sharding to balance load and reduce latency.
- Relevance Ranking Engines - Implements a relevance ranking engine using BM25, term frequency, and inverse document frequency algorithms.
- Complex Search Querying - Executes complex queries utilizing logical operators, token proximity, and phonetic conversions for advanced retrieval.
- Search Result Aggregators - Provides a framework for collecting and normalizing search results from sharded servers into a single ranked result set.
- Text Segmentation - Splits textual content into discrete keywords using various participle methods to optimize indexing.
- Chinese Language Segmenters - Includes concurrent Chinese text segmentation to tokenize Chinese strings for efficient indexing and retrieval.
- Hash Slot Sharding - Uses hash-based sharding to distribute the search index across multiple server nodes for load balancing.
- Distributed Index Sharding - Partitions large search indexes across multiple servers to manage datasets exceeding single-node memory.
- Distributed Text Indexing - Shards large textual datasets across servers to enable low-latency queries and high-throughput retrieval.
- BM25+ Scoring - Implements the BM25 algorithm for calculating document relevance based on term frequency and inverse document frequency.
- Real-time Indexing - Supports real-time addition and removal of documents while the engine is running to maintain continuous availability.
- Boolean Logic Engines - Provides a logic engine for executing boolean queries and evaluating token proximity within the index.
- Data Persistence Management - Manages the lifecycle of writing in-memory index updates to disk to prevent data loss.
- Persistence & Durability - Ensures index durability across system restarts by writing data to persistent storage backends.
- In-Memory Databases with Persistence - Combines high-performance in-memory indexing with persistent disk storage to ensure data durability.
- Phonetic Pinyin Mappings - Implements phonetic Pinyin mapping to allow romanized input to match Chinese characters during retrieval.
- Index Lifecycle Management - Manages the index lifecycle by allowing real-time additions and removals of documents without requiring full rebuilds.
- Custom Scoring Multipliers - Allows the use of custom scoring logic and data fields to influence the relevance ranking of documents.
- Index State Commitments - Ensures data durability by committing pending index changes to persistent storage.
- Search Ranking Algorithms - Uses heuristic-based systems and custom rating fields to refine and prioritize the order of search results.
- Proximity Filtering - Filters results by measuring the distance between keywords to ensure terms appear within a specified proximity.
- Cloud-Based Indexing Services - Provides a backend indexing server that processes textual data into searchable indexes with real-time updates.
- Search and Recommendation - Full-text search engine.
- Search and Recommendation - Full-text search engine.
- Search Engines - Distributed, efficient search engine.
