1 Repo
Awesome GitHub RepositoriesTokenized
Local persistence of pre-processed and tokenized datasets to accelerate training restarts.
Distinct from Data Caching: Specifically caches tokenized ML datasets rather than general application state or session data.
Explore 1 awesome GitHub repository matching data & databases · Tokenized. Refine with filters or upvote what's useful.
Awesome Tokenized GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- allenai/open-instruct
allenai/open-instruct
3,586Auf GitHub ansehenOpen-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a coordinator for supervised fine-tuning, reinforcement learning from human feedback pipelines, and tool-use training, providing specialized roles for dataset curation and model alignment. The project distinguishes itself through a high-performance training architecture that utilizes actor-based distributed coordination and hybrid sharding to manage large GPU clusters. It implements advanced alignment techniques including direct preference optimization, group relative policy opt
Stores processed and concatenated datasets on disk to prevent redundant computation during distributed training restarts.
Python
