zihangdai/xlnet
Xlnet
This project is a natural language processing framework focused on a generalized autoregressive pretrainer designed for unsupervised language representation. It implements a language model that combines permutation-based training with a Transformer-XL backbone to function as a long-context text processor.
The system is distinguished by its ability to handle text sequences that exceed standard length limits through the use of segment-level recurrence and relative positional encoding. It scales high-performance pretraining across multiple GPUs and TPU clusters using distributed training implementations.
The codebase covers the full machine learning workflow, including text cleansing and subword tokenization for data preprocessing, as well as task-specific fine-tuning for question answering, reading comprehension, and text classification. It includes utilities for parameter optimization, learning rate scheduling, and the evaluation of answer probabilities through precision-recall metrics.
The project provides configurations for managing model hyperparameters and hardware-accelerated training across multiple hosts.
Features
- Permutation Language Models - Implements a generalized autoregressive pretrainer combining autoregressive modeling with permutation-based training.
- Recurrent Transformer Architectures - Implements a Transformer-XL backbone that combines attention with segment-level recurrence for extended context processing.
- Language Representations - Learns general semantic and syntactic patterns from unlabeled text for use in downstream tasks.
- Long-Context Models - Designed to handle text sequences exceeding standard length limits through segment-level recurrence.
- Long Context Processing - Processes and analyzes sequences of text that exceed standard length limits by managing long-range dependencies.
- Distributed Training - Scales the training of large-scale language models by distributing workloads across multiple GPU and TPU hosts.
- Language Model Pretraining - Trains a generalized autoregressive model on large text corpora using hardware accelerators.
- Segment-Level Recurrence - Caches hidden states from previous segments to maintain long-range dependencies across sequences exceeding a fixed window.
- Relative - Tracks the distance between tokens to better generalize to sequence lengths not encountered during training.
- Transformer Language Models - Uses a Transformer-XL backbone to learn unsupervised language representations.
- Unsupervised Pre-training - Implements a generalized autoregressive pretrainer for unsupervised language representation on large text corpora.
- Permutation Language Modeling - Implements a generalized autoregressive pretraining objective using all possible permutations of the input sequence.
- Distributed Training Optimizers - Accelerates model training by distributing workloads across multiple hardware hosts and cores.
- Hyperparameter Configurations - Provides configurations for layer counts, attention heads, and hidden sizes to ensure consistency across training phases.
- Warmup Schedules - Uses linear warmup schedules to gradually increase the learning rate and stabilize initial unsupervised learning.
- GPU-Accelerated Training - Executes unsupervised pretraining of language representation models using GPU acceleration.
- TPU Training Accelerators - Manages cluster resolvers and run configurations to enable hardware-accelerated training on TPU clusters.
- Classification Fine-Tuning - Adapts pretrained models for single-document and document-pair classification and regression tasks.
- Synchronous Gradient Averaging - Synchronizes model updates across multiple hardware accelerators by computing the mean of gradients across training replicas.
- Natural Language Processing - Provides a framework for training and fine-tuning language representations for various NLP tasks.
- Question Answering Fine-tuning - Fine-tunes models on reading comprehension datasets to extract precise answers from long-context passages.
- Question Answering Systems - Fine-tunes a language model to extract precise answers from text passages and evaluate reading comprehension accuracy.
- Subword Tokenization - Segments raw text into subword units while applying specialized rules for numerical punctuation.
- Text Tokenizers - Transforms raw text into a sequence of integer identifiers by tokenizing input and mapping pieces to IDs.
- Vocabulary Construction - Transforms raw text into numerical IDs using a vocabulary model for data processing.
- Reading Comprehension - Implements training processes to map text positions for answering questions within provided passages.
- Language Modeling - Generalized autoregressive pre-training for language understanding.
- Natural Language Processing - Autoregressive pretraining for language understanding.
- Transformer Implementations - Official code repository for the XLNet research paper.
