google-deepmind/learning-to-learn
Learning To Learn
Acest proiect este un framework de meta-learning TensorFlow și un set de instrumente de cercetare conceput pentru a implementa și antrena optimizatori învățați. Oferă o bibliotecă de instrumente pentru dezvoltarea rețelelor neuronale care învață cum să optimizeze alte modele, înlocuind algoritmii tradiționali de optimizare bazați pe gradient.
Framework-ul include un manager de ansamblu de probleme care permite combinarea mai multor sarcini de optimizare distincte într-o singură funcție de pierdere ponderată pentru antrenare simultană. Utilizează un model factory pentru instanțierea rețelei și suportă definirea funcțiilor obiectiv personalizate și a grafurilor de pierdere ca ținte pentru algoritmii de învățare.
Setul de instrumente acoperă o gamă largă de capabilități, inclusiv meta-optimizarea bazată pe gradient, benchmarking-ul modelelor și execuția buclelor de antrenare cu lungimi de derulare configurabile. De asemenea, oferă utilitare pentru preprocesarea gradientului, persistența stării serializate și raportarea statisticilor experimentelor, cum ar fi eroarea finală medie și durata epocii.
Features
- Meta-Optimization - Uses gradient-based meta-optimization to refine the learning process of target neural networks.
- Deep Learning Toolkits - Provides a set of utilities for configuring problem architectures and evaluating convergence on benchmark datasets.
- Problem Ensembling - Manages the combination of multiple distinct optimization tasks into a single weighted loss function.
- Learned Optimizers - Implements a library of tools for building and training neural networks that act as optimizers.
- Loss Graph Definitions - Provides a way to define mathematical tasks and loss graphs as targets for learned optimization algorithms.
- Custom Loss Functions - Implements custom objective functions and weighted problem ensembles to calculate error during training.
- Meta Learning Frameworks - Provides a TensorFlow-based framework for training neural networks to optimize other models.
- Learned Optimizer Training - Implements training loops with configurable epochs and unroll lengths to improve the performance of learned optimizers.
- Meta-Optimizer Training - Executes training loops that optimize a learning algorithm based on cumulative evaluation costs over specified epochs.
- Problem Ensembling - Combines multiple distinct optimization tasks into a single weighted loss function for simultaneous training.
- Meta Learning Libraries - Provides a toolkit for developing and training neural networks that learn how to optimize other models.
- Problem Ensemble Composition - Combines multiple distinct optimization tasks into a single weighted loss function for simultaneous training.
- Data Preprocessing - Transforms input gradients using logarithmic scaling and sign extraction to prepare them for model consumption.
- Gradient Distribution Transformations - Applies logarithmic transformations and preserves input signs to reshape the distribution of processing data.
- Meta-Learning Problem Definitions - Enables the creation of loss graphs for mathematical tasks to serve as training targets for learning algorithms.
- Gradient Flow Stabilizers - Applies logarithmic scaling and sign extraction to gradients to ensure numerical stability during training.
- Optimizer Convergence Evaluators - Provides utilities to test optimizer effectiveness by measuring convergence and performance on target problems.
- Model Execution Benchmarks - Evaluates the execution cost and performance of optimization algorithms across diverse mathematical problem sets.
- Benchmark Problem Configurations - Provides tools for setting up network architectures and hyperparameters for consistent testing across benchmark tasks.
- Problem Modeling - Provides a framework for defining optimization problem structures, including custom objective functions and loss operations.
- Unroll-Length Optimizations - Executes training iterations over specified sequence lengths to optimize the learning algorithm's performance.
