Autoresearch is an autonomous machine learning research agent and architecture search framework. It employs a closed-loop system to programmatically rewrite training and architecture source code to discover optimal language model configurations.
The system iteratively modifies code and evaluates performance metrics to improve model quality based on a target objective. It optimizes model performance and training efficiency by tracking validation bits per byte, which allows for a fair comparison of architectural changes independently of vocabulary size.
The framework manages the full training workflow on a single GPU, utilizing Git branches to isolate experimental changes and version track successful improvements. It incorporates fixed-budget time constraints for each run and maintains structured logs of performance metrics and memory usage across all trials.
