thunlp/OpenPrompt
OpenPrompt
OpenPrompt es un framework de aprendizaje de prompts diseñado para adaptar modelos de lenguaje de gran tamaño (LLM) a tareas de procesamiento de lenguaje natural (NLP) posteriores. Proporciona un kit de herramientas integral para implementar estrategias de prompting manual, suave (soft) y continuo, permitiendo refinar los modelos sin actualizar todos los parámetros subyacentes.
El proyecto se distingue por su soporte tanto para el ajuste de prompts discretos como continuos. Incluye un sistema para inyectar tokens y embeddings entrenables en las entradas del modelo mediante descenso de gradiente, así como un motor de generación automática de prompts que utiliza búsqueda de haz (beam search) y modelos generativos para descubrir plantillas de texto de alta probabilidad para conjuntos de datos específicos.
El framework cubre varias áreas de capacidad principales, incluyendo el diseño de plantillas y la verbalización de etiquetas para mapear etiquetas de clasificación a palabras del vocabulario. También proporciona herramientas de adaptación de modelos para envolver modelos preentrenados, calibración de logits para mejorar la precisión de las predicciones y un pipeline de datos que incluye lógica de muestreo especializada para el aprendizaje con pocos ejemplos (few-shot learning).
Los flujos de trabajo de entrenamiento y experimentación se gestionan a través de archivos de configuración que definen escenarios de aprendizaje, hiperparámetros y especificaciones del pipeline.
Features
- Prompt-Based Adaptation - Adapts pre-trained models to specific tasks using textual templates and label verbalizers without updating all model parameters.
- Prompt Learning Frameworks - Offers a framework for adapting large language models to downstream tasks using textual templates and label verbalizers.
- Continuous Prompt Tuning - Integrates trainable soft tokens and continuous embeddings into text templates to optimize performance via gradient descent.
- Verbal Label Projections - Projects original class labels onto specific target vocabulary words to facilitate verbalized predictions.
- Few-Shot Learning Frameworks - Provides a comprehensive infrastructure for performing NLP classification and entity extraction with minimal labeled examples.
- Few-Shot Text Learning - Trains language models to perform classification or generation tasks using only a very small number of labeled examples.
- Label Verbalizers - Projects classification labels onto specific vocabulary words to translate model logits into human-readable category predictions.
- NLP Template Engines - Includes a system for defining and managing textual patterns and placeholders to structure inputs for language models.
- Prompt-Based Model Adaptation - Adapts pre-trained models by wrapping them in specialized layers that handle prompt formatting and label projection.
- Template Definitions - Inserts placeholders and masks into original input text to guide model behavior for specific tasks.
- Prompt Template Transformations - Wraps raw text in structured patterns with placeholders and masks to adapt pre-trained models to specific tasks.
- Prompt Templates - Defines textual patterns and placeholder mappings to implement manual or prefix-tuning strategies for model inputs.
- Prompt Tuning Libraries - Provides a library of manual, soft, and continuous prompting strategies to refine model behavior without full parameter updates.
- Vocabulary Projection - Projects target classification labels onto specific words within the model vocabulary to enable verbal predictions.
- Automatic Label Verbalization - Finds optimal vocabulary words to represent classification labels using scoring functions and training data.
- NLP Model Assemblers - Assembles text classifiers by combining modular components for model architecture, data formats, and tensorizers.
- Prompt Text Preparations - Wraps datasets with templates and tokenizes them through a pipeline to manage sequence lengths.
- Few-Shot Sample Selection - Selects small subsets of data based on total counts or per-class quotas for few-shot learning.
- Few-Shot Learning Optimizers - Automatically generates optimal templates and verbalizers to improve performance in low-resource few-shot scenarios.
- Beam Search Implementations - Implements beam search algorithms to automatically discover high-probability text templates for specific datasets.
- Prompt Pipeline Encapsulations - Combines language models, text templates, and verbalizers into a single unified object for streamlined training and inference.
- Logit Calibration - Calculates baseline logit tensors across the vocabulary to adjust prediction probabilities and improve classification accuracy.
- Automatic Template Discovery - Uses beam search and generative models to automatically discover high-probability text templates for specific datasets.
- Automatic Template Generation - Creates high-probability text templates using beam search decoding based on a dataset and generation model.
- Zero and Few-Shot Learning - Configures operation modes for full-supervision, zero-shot, and few-shot learning scenarios.
- Few-Shot Sample Selection - Provides logic to select a minimal subset of training data based on class quotas for low-resource learning.
- Label Projection Learning - Optimizes embedding weights used to project classification labels onto the model vocabulary.
- Prompted Data Loaders - Integrates tokenization and template application directly into the training pipeline using specialized data loaders.
- Prompt Improvement Methods - Prototypical verbalizer implementation for few-shot tuning.
- Prompt Learning - Modular framework for prompt-learning and task-specific adaptation.
- Prompt Learning Frameworks - A comprehensive framework for prompt-learning research and implementation.
- Prompt Optimization Frameworks - Framework for research into prompt-learning techniques.
