facebookresearch/LASERArchived
LASER
LASER is a cross-lingual sentence embedding library and multilingual text encoder. It functions as a parallel text mining tool that maps sentences from multiple languages into a shared vector space for similarity and classification tasks.
The system converts raw text into fixed-length embeddings, enabling the discovery of translation pairs by calculating the vector distance between sentences. This shared representation allows for cross-lingual document classification, where a model trained on one language can be used to categorize documents in another.
The library includes a sentence-piece tokenizer to split multilingual strings into subword units. It also provides mechanisms for pre-trained model management, allowing users to download and store language models locally for offline embedding generation.
Features
- Cross-Lingual Alignment - Provides a shared vector space to compare meanings across different languages regardless of the original script.
- Shared Latent Spaces - Maps sentences from multiple languages into a single coordinate system for semantic comparison.
- Multilingual NLP - Prepares multilingual text for machine learning tasks through tokenization and model management.
- Multilingual Encodings - Converts raw text into fixed-length embeddings using pre-trained multilingual models.
- Sentence Embeddings - Maps sentences from multiple languages into a shared vector space for similarity and classification.
- Cross-Lingual Transfer - Enables classification tasks in one language using a model trained in another via shared embeddings.
- Fixed-Length Vector Encodings - Transforms variable-length text sequences into single constant-size vectors for efficient similarity comparisons.
- Text Tokenization - Splits raw text into smaller units using sentence-piece models for encoding.
- Subword Tokenization - Uses subword tokenization to handle unseen words and diverse language scripts.
- Vector Similarity Search - Calculates angular distance between high-dimensional vectors to identify translation pairs.
- Parallel Corpus Mining - Discovers translation pairs across different languages by calculating vector distance between embeddings.
- Cross-Lingual - Assigns categories to documents in one language using a model trained on another via shared embeddings.
- Natural Language Processing - Library for computing and using multilingual sentence embeddings.
- Natural Language Corpora - Massive multilingual parallel text corpus for cross-lingual tasks.
