google/sentencepiece

Features

• Subword Tokenization - Trains and applies subword segmentation models using Unigram or BPE algorithms for natural language processing.
• Text Tokenizers - Transforms raw text into subword units or numerical identifiers using trained segmentation models.
• Natural Language Processing - Converts raw text into subword units or numerical identifiers to prepare data for large language models.
• Text Segmentation - Splits text into subword pieces with support for byte-level fallback and stochastic sampling.
• Special Symbol Managers - The tokenization library customizes the surface strings and integer identifiers for reserved tokens like unknown, beginning-of-sequence, end-of-sequence, and padding markers.
• Machine Learning Training - Transforms raw text into numerical identifiers and manages vocabulary constraints for neural network training.
• Natural Language Processing Libraries - Provides a collection of tools for normalizing, encoding, and decoding text into subword units.
• Byte-Level Tokenizers - Decomposes unknown characters into UTF-8 byte sequences to ensure full vocabulary coverage without unknown tokens.
• Data Preprocessing - Cleans and normalizes text inputs while managing vocabulary constraints for neural network models.
• Byte Level Encodings - Ensures full vocabulary coverage by treating raw input as a sequence of bytes rather than Unicode characters.
• Subword Regularization Methods - Improves model robustness during training by sampling multiple possible token sequences for a single input.
• Token Alignment Trackers - Maps individual tokens back to their original character or byte positions for precise text alignment and extraction.
• Segmentation Model Training - Creates new tokenization models from raw text data using flexible input sources.
• Vocabulary Management - Provides tools for defining specialized control symbols and atomic tokens to handle domain-specific requirements.
• Token Decoders - Reconstructs original raw text from sequences of subword pieces or numerical identifiers.
• Vocabulary Management - Applies rules on subword length and character boundaries to control the structure and composition of the generated vocabulary.
• Natural Language Processing - Language-independent subword tokenizer and detokenizer.
• Segmentation Boundary Enforcers - Enforces hard segmentation boundaries using custom delimiters to influence tokenization logic.
• Memory-Mapped Weight Loaders - Maps model weight files directly into process memory to reduce RAM usage and improve load times.
• Vocabulary Usage Restrictions - Limits tokenization output to a specific subset of allowed symbols to control the active vocabulary during inference.
• High-Performance and Parallel Computing - Distributes tokenization and detokenization tasks across multiple CPU threads to rapidly handle massive datasets.
• Parallel Processing - Distributes encoding and decoding workloads across multiple CPU threads to increase processing speed.
• Control Symbol Reservoirs - Reserves specific vocabulary identifiers for model control flow that do not participate in text segmentation.
• Atomic Token Definitions - Preserves the integrity of specific character sequences by treating them as indivisible tokens during encoding.
• Language Model Architectures - Uses probabilistic models to determine the most likely subword segmentation by evaluating token combinations.
• Model Training Optimizers - Controls corpus loading, shuffling, and character coverage thresholds to optimize model training.
• Segmentation Restriction Utilities - Enforces constraints on token formation to ensure valid vocabulary generation.
• Whitespace Formatting Utilities - Configures whitespace handling rules to ensure consistent text representation during the tokenization process.
• Text Normalization - Applies Unicode normalization rules and custom whitespace handling to ensure consistent text representation.
• Unicode Normalization Pipelines - Applies Unicode transformations and whitespace rules to ensure consistent text representation before segmentation.

Star history