Awesome GitHub RepositoriesTokenization Pipelines

lunr.js is a JavaScript full-text search library and client-side search engine. It creates in-memory search indexes for fast keyword retrieval and ranked document matching within browser or Node.js environments. The library utilizes a JSON serializable search index, allowing the search structure to be converted to and from JSON for storage and distribution of pre-built search data. This enables search functionality for static websites by indexing content into portable files. The system supports advanced querying capabilities, including fuzzy text matching to account for typos, field-scoped i

Vespa is a distributed search engine, vector database, and machine learning ranking engine. It serves as an AI search platform designed to handle large-scale document indexing and complex query processing across a cluster of nodes, combining keyword retrieval with high-dimensional embedding storage for semantic similarity search. The platform distinguishes itself by integrating machine learning models directly into the search pipeline to perform real-time inference and ranking. It converts these models into ranking expressions to score and order results based on relevance, while providing a s

Zombodb is a database extension and relational data indexer that integrates PostgreSQL with Elasticsearch. It provides a SQL search interface, allowing users to execute complex search queries and aggregations using standard SQL functions and syntax instead of native JSON APIs. The project synchronizes relational data from PostgreSQL to a remote search engine to enable high-performance full-text search and analytics. The system distinguishes itself by bridging relational structures with search engine capabilities, specifically through geospatial search integration for geometry and geography ty

WhisperSpeech هو نظام تحويل الكلام إلى نص متعدد اللغات ونظام تحويل النص إلى كلام عصبي. يعمل عن طريق عكس بنية نموذج Whisper لتحويل النص إلى صوت اصطناعي عالي الدقة. يتيح النظام استنساخ الصوت باستخدام ملفات صوتية مرجعية لتقليد متحدثين محددين. يدعم إنتاج الكلام متعدد اللغات، والذي يتضمن القدرة على توليد الصوت عبر لغات مختلفة والتعامل مع تبديل اللغة داخل جملة واحدة. يغطي المشروع مجموعة واسعة من قدرات الكلام، بما في ذلك توليد النص إلى كلام وإعداد مجموعة بيانات الكلام. يتضمن أدوات لنسخ الكلام إلى نص، واستخراج الرموز الصوتية، واكتشاف نشاط الصوت.

Bark Voice Cloning is a text-to-speech synthesis engine designed to generate natural-sounding audio and replicate specific vocal characteristics. The system utilizes a transformer-based autoregressive model to convert written text into high-fidelity speech, supporting multilingual output and expressive delivery. The project distinguishes itself through zero-shot voice cloning, which extracts speaker identity embeddings from short audio samples to condition the generative model without requiring extensive fine-tuning. It also provides specialized workflows for voice identity conversion, allowi

This project is a training pipeline and framework for developing Chinese language models based on the Llama 2 architecture. It functions as a distributed GPU trainer and dataset preprocessing toolkit designed for both the initial pre-training of baseline models and subsequent supervised fine-tuning. The system distinguishes itself through a specialized workflow for Chinese text, incorporating a data curation pipeline that uses similarity hashing for deduplication and a tokenization process that converts raw text into memory-mapped binary files for efficient disk access. It implements a superv