Jiayi-Pan/TinyZero
TinyZero
TinyZero is a reinforcement learning framework and implementation designed to train language models to develop reasoning and self-verification abilities. It provides a training pipeline to optimize model performance on mathematical and logical tasks.
The project serves as a minimal reproduction of the DeepSeek R1 architectural and training approach. It focuses on creating reasoning models that can solve structured problems through autonomous chain-of-thought discovery.
The framework incorporates group relative policy optimization and reward-based self-correction to improve accuracy on logical tasks such as counting and multiplication. These capabilities are supported by Monte Carlo rollouts and a reward system based on final answer accuracy and format correctness.
Features
- Reinforcement Learning Alignment - Implements a reinforcement learning alignment framework using Group Relative Policy Optimization (GRPO).
- Chain-of-Thought Prompting - Utilizes chain-of-thought prompting to enable the model to generate intermediate reasoning steps.
- Model Fine-Tuning - Provides procedures for fine-tuning models to solve structured logical and mathematical problems.
- Self-Correction Loops - Provides self-correction loops that use reward signals to refine internal reasoning logic.
- Reasoning Model Training Suites - Provides a training suite designed to develop self-verification and logic abilities in language models.
- Reasoning Models - Implements a reasoning model optimized for complex logical deduction and multi-step processing.
- Reinforcement Learning Optimizers - Optimizes model policies using reward-based techniques to improve performance on mathematical tasks.
- Reinforcement Learning Training Pipelines - Ships a training pipeline that orchestrates the lifecycle of reinforcement learning for logical tasks.
- Reinforcement Learning Training Utilities - Includes utilities for training model weights via reinforcement learning to improve logical task accuracy.
- Replications - Serves as a minimal reproduction of the DeepSeek R1 architectural and training approach.
- Monte Carlo Trajectory Estimators - Uses Monte Carlo trajectory estimators to evaluate multiple completion paths for logical accuracy.
- Model Implementations - Minimal reproduction of reasoning-focused language models.
- Reasoning Models - Minimalist implementation of reasoning-focused training.
- Reinforcement Learning Frameworks - Efficient reinforcement learning implementation for small reasoning models.
