1 repositorio
Awesome GitHub RepositoriesTokenized
Local persistence of pre-processed and tokenized datasets to accelerate training restarts.
Distinct from Data Caching: Specifically caches tokenized ML datasets rather than general application state or session data.
Explore 1 awesome GitHub repository matching data & databases · Tokenized. Refine with filters or upvote what's useful.
Awesome Tokenized GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- allenai/open-instruct
allenai/open-instruct
3,586Ver en GitHubOpen-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a coordinator for supervised fine-tuning, reinforcement learning from human feedback pipelines, and tool-use training, providing specialized roles for dataset curation and model alignment. The project distinguishes itself through a high-performance training architecture that utilizes actor-based distributed coordination and hybrid sharding to manage large GPU clusters. It implements advanced alignment techniques including direct preference optimization, group relative policy opt
Stores processed and concatenated datasets on disk to prevent redundant computation during distributed training restarts.
Python
