facebookresearch/fairseqArchived

Features

• Language Model Training - Trains models to predict the next token in a sequence using configurable architectures and optimizers.
• Neural Machine Translation - Translates text or speech between multiple languages using sequence-to-sequence models and multilingual embeddings.
• Sequence-to-Sequence Tasks - Builds and optimizes models designed to transform input sequences into output sequences for diverse tasks.
• Neural Machine Translation Frameworks - Provides a complete system for building multilingual translation models with tokenization, beam search, and data mining.
• Data Preprocessing Pipelines - Transforms raw text into training-ready formats by applying tokenization and creating language-specific dictionaries.
• Distributed Model State Management - Implements parameter sharding and CPU offloading to handle large-scale models that exceed single-device memory.
• Data-Parallel Training - Distributes training workloads across multiple GPUs by replicating model states and sharding training batches.
• Distributed Training - Provides a framework for training large language models using mixed precision, sharded parameters, and CPU offloading.
• Distributed Training Sharding - Implements strategies for partitioning model parameters and optimizer states across multiple GPUs to enable training of massive models.
• Large-Scale Model Training - Manages distributed training workflows across multiple GPUs using sharding, mixed precision, and memory offloading.
• Dataset Preprocessing Utilities - Converts raw text files into an optimized binary format by applying padding and token thresholds.
• Mixed Precision Training - Utilizes a combination of 16-bit and 32-bit floating point formats to accelerate training and reduce memory consumption.
• Model Fine-Tuning - Adjusts pre-trained model weights on specific datasets to improve performance on specialized tasks.
• Pre-trained Model Zoos - Loads ready-to-use models for translation and language modeling via a standardized hub interface.
• Model Parallelism - Provides the ability to split model parameters across multiple GPUs to train models that exceed single-device memory.
• Byte Pair Encodings - Processes raw text or sentence pairs into byte-pair encoding tokens and decodes them back to text.
• Automatic Speech Recognition - Converts spoken audio recordings into written text using pre-trained automatic speech recognition models.
• Self-Supervised Speech Representations - Learns linguistic speech representations from unlabeled multilingual audio data using self-supervised learning.
• Acoustic Feature Processing - Transcribes audio signals into written text by processing acoustic features through sequence-based networks.
• Model Weight Management - Imports weights and configurations for existing sequence models to perform inference or fine-tuning.
• Multi-GPU Training Utilities - Provides utilities for distributing translation and language model training workloads across multiple GPUs.
• Multilingual Parameter Sharing - Develops shared networks for multiple language pairs using unified parameters and layer selection patterns.
• Multilingual Text Processing - Maps text from one language to another using multilingual models and diverse translation methods.
• Natural Language Processing - Provides preprocessing pipelines that transform raw text via tokenization and binarization for model training.
• Speech-to-Text Modeling Toolkits - Provides a specialized toolkit for pre-training and fine-tuning speech representations for audio transcription and translation.
• Speech Transcription - Produces phone or word labels from speech representations using trained unsupervised models.
• Text Tokenizers - Converts raw text into integer sequences using Byte-Pair Encoding and provides decoding back to text.
• Multilingual Encodings - Converts text from various languages into numerical tokens using a sentence-piece model.
• Translation Quality Metrics - Produces translations and calculates standardized metrics like BLEU to evaluate output quality.
• From-Scratch Training - Trains models from scratch using custom datasets to learn general language representations.
• Sequence To Sequence Models - Provides transformer-based sequence-to-sequence models for translating text between multiple languages.
• Text Sequence Generators - Produces text or speech representations using advanced search algorithms like beam search and nucleus sampling.
• Binarized - Processes raw text and alignment files into a binary format for efficient loading during training.
• Distributed Parameter Sharding - Distributes model parameters and optimizer states across multiple devices to bypass single-GPU memory limits.
• Dataset Binarization - Binarizes raw text datasets into a format optimized for efficient loading during model training and evaluation.
• Audio Tokenization - Transforms raw audio signals into discrete numerical tokens using vector quantization for sequence-based modeling.
• Back-Translation Data Augmentation - Creates synthetic parallel data by translating monolingual text back into the source language to improve model performance.
• Dynamic Convolutions - Implements lightweight dynamic convolution architectures to reduce computational overhead in sequence-to-sequence operations.
• Denoising Auto-Encoders - Trains a denoising auto-encoder on large-scale monolingual corpora to create a starting point for downstream tasks.
• Feature Extraction Models - Retrieves hidden state representations from model layers to use as embeddings for downstream tasks.
• Token Embedding Extraction - Generates numerical vector representations of tokens from model layers for detailed downstream analysis.
• Hidden State Extraction - Retrieves feature vectors from the final layer or all hidden layers for a given set of tokens.
• Text Embedding Extraction - Retrieves hidden state features from model layers to represent text numerically.
• Output Constraint Engines - Forces the model to include specific tokens or phrases in its output during decoding via constraint engines.
• Speech Synthesis - Generates audible speech from textual input by predicting acoustic waveforms from sequence data.
• Kernel Optimizations - Uses specialized CUDA kernels for convolution operators to significantly reduce memory usage during long sequence processing.
• Spoken Language Detection - Classifies the language spoken in an audio sample by extracting embeddings and calculating accuracy.
• Long Context Training Optimizations - Implements truncated backpropagation through time to manage memory and context when training on long sequences.
• Perplexity Calculators - Computes model perplexity based on cross-entropy loss to measure predictive performance on test datasets.
• Speech Model Fine-Tuning - Optimizes pretrained speech models using Connectionist Temporal Classification (CTC) to convert audio to text.
• Mixture of Experts - Supports training sequence-to-sequence models using mixture-of-experts architectures with learned priors.
• Expert-Based Generation - Produces translation output by specifying a particular expert from a trained mixture model to handle generation.
• Byte Level Encodings - Implements byte-level byte-pair encoding to handle open vocabularies and eliminate unknown tokens.
• Speech Representation Learning - Trains models on unlabeled multilingual speech data to extract general-purpose audio features.
• Performance Evaluation - Measures speech model performance using external language models and Viterbi decoding for transcription verification.
• Beam Search Decoders - Implements beam search algorithms to maintain multiple candidate sequences during the decoding phase of sequence generation.
• Convolutional Architectures - Implements a convolutional architecture for sequence-to-sequence tasks to translate text between languages.
• Speech Translation Systems - Adapts pre-trained speech models to translate spoken audio from one language into text in another.
• Model Depth Optimization - Implements structured dropout to allow on-demand reduction of encoder or decoder depth.
• Model Pruning - Removes redundant layers from trained models to create smaller and more efficient versions.
• Model Quantization - Applies iterative product quantization to reduce the memory footprint of pre-trained models.
• Quantization-Aware Training - Trains models using quantization noise to prepare them for extreme compression via product quantization.
• Multilingual Representation Learners - Generates language-agnostic sentence representations to enable semantic comparisons across different languages.
• Multimodal Input Processors - Provides a framework for ingesting and aligning diverse data sources like video, audio, and text using contrastive learning.
• Multimodal Machine Learning - Processes and aligns disparate data sources such as video, audio, and text for classification and retrieval.
• Multimodal Processing - Provides frameworks for aligning and integrating diverse data modalities like video and text within shared embedding spaces.
• Text Summarization - Utilizes sequence-to-sequence models to generate concise summaries of long-form documents.
• Pointer-Generator Networks - Uses a pointer-generator network to copy words from input to output to handle out-of-vocabulary terms.
• Evaluation Metrics - Provides standardized accuracy and quality metrics to evaluate the performance of trained sequence-to-sequence models.
• Linear-Time Sequence Models - Provides self-attention mechanisms with linear complexity to efficiently process long sequences.
• Transcription Refinement Pipelines - Improves transcription quality through an iterative process of generating pseudo-labels and fine-tuning with CTC loss.
• Supervised Fine-Tuning - Adapts pre-trained sequence-to-sequence models to specific language pairs using supervised datasets.
• Text Generation - Provides a pipeline for generating natural language text sequences from trained language models.
• Text Generation Strategies - Implements sampling and decoding strategies including beam search and temperature control for text generation.
• Masked - Fills missing words in a text sequence by predicting candidates based on surrounding context.
• Text-to-Speech - Synthesizes natural human speech waveforms from input text strings using trained generator checkpoints.
• Speech-to-Speech Models - Implements end-to-end conversion of spoken audio from one language directly into spoken audio of another.
• Iterative Data Mining - Improves cross-lingual alignment by alternating between mining pseudo-parallel data and training.
• Non-Autoregressive Training - Implements training for sequence-to-sequence models that generate output in parallel instead of token-by-token.
• Parallel Corpus Mining - Implements algorithms to identify semantically similar sentence pairs across different languages for unsupervised learning.
• Multimodal Data Handlers - Aligns disparate input sources like video and text using decoupled processors to simplify data loading.
• Distributed Tensor Sharding - Employs sharded tensors and memory-mapped arrays to optimize input-output overhead during distributed training.
• Model Layer Offloading - Moves optimizer states and model copies to system memory to lower VRAM usage for large-scale models.
• Audio Feature Extraction - Extracts numerical vector representations from raw audio data using trained models for downstream machine learning tasks.
• Real-time Translation - Translates text or speech in real-time using wait-k or monotonic multihead attention strategies.
• Model Evaluation - Measures the predictive accuracy of language models using perplexity metrics on specific datasets.
• Cross-Modal Models - Contrastive pre-training for zero-shot video-text understanding.
• Dynamic Networks - Lightweight and dynamic convolutions for sequence modeling.
• Generative Models - Sequence-to-sequence learning toolkit for translation and language tasks.
• Natural Language Processing - Convolutional sequence-to-sequence learning framework.
• Open Source Models - Toolkit for sequence modeling and machine translation research.
• Perception Models - Robustly optimized approach for BERT-style pre-training.
• Self-Supervised Pretraining - General framework for self-supervised learning across modalities.
• Training and Orchestration - Toolkit for training sequence modeling and generation tasks.

Historial de estrellas