ML-GSAI/LLaDA
LLaDA
LLaDA is a masked diffusion language model and conditional text generator. It generates text by iteratively refining masked tokens through a diffusion process rather than predicting the next token in a sequence.
The project functions as a vision-language diffusion model, converting visual inputs into text responses. It also serves as a preference optimization framework that uses log-likelihood estimation and evidence lower bounds to tune model responses.
The system supports multi-round conversational AI and text sequence evaluation. It integrates vision-language embedding for cross-modal conditioning and uses iterative token refinement to produce text.
Features
- Masked Text Diffusion - Implements a masked diffusion architecture that iteratively refines tokens into final text.
- Multimodal Diffusion Models - A system that converts visual inputs into text responses using a diffusion process for complex multimodal tasks.
- Vision-Language Cross-Attention Fusions - Fuses visual encoder features with text embeddings to enable image-conditioned text generation.
- Cross-Attention Conditioning - Implements cross-attention mechanisms to inject visual and textual context into the diffusion denoising process.
- Iterative Refinement Generation - Generates text by repeatedly updating and refining masked sequences until convergence.
- Vision-Language Models - Integrates visual and linguistic processing to describe images and answer questions.
- Diffusion Masked Language Models - Generates text by iteratively refining masked tokens via a diffusion process.
- Preference Optimization - Refines model responses based on human preferences using log-likelihood and evidence lower bounds.
- Visual-to-Text Generation - Converts visual inputs into text responses using a diffusion process for multimodal tasks.
- Discrete Diffusion Models - Utilizes a discrete state space and categorical distributions for token transitions during diffusion.
- Variational Lower Bound Estimations - Calculates likelihoods via a variational lower bound to optimize model responses for specific preferences.
- Conditional Masked Generators - Produces conditional text sequences by iteratively filling in missing tokens through masking.
- Sequence Likelihood Estimators - Calculates probability estimates of sequences using evidence lower bounds for preference optimization.
- Sequence Likelihood Scores - Measures the log-likelihood of text sequences to evaluate model predictive accuracy.
- Language Diffusion Models - Large language diffusion models for text generation.
