1 个仓库
Awesome GitHub RepositoriesDataset Binarization
Processes of converting text datasets into binary formats to optimize ingestion speed.
Distinct from Data & Text Processing: Focuses on the binary conversion process for model loading rather than general text processing utilities.
Explore 1 awesome GitHub repository matching programming languages & runtimes · Dataset Binarization. Refine with filters or upvote what's useful.
Awesome Dataset Binarization GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- facebookresearch/fairseq
facebookresearch/fairseq
32,228在 GitHub 上查看Fairseq is a PyTorch toolkit for sequence-to-sequence modeling, specializing in neural machine translation, automatic speech recognition, and large-scale language model training. It provides a framework for processing and aligning diverse data sources, including text, audio, and video, to support tasks such as speech-to-text conversion and multimodal sequence learning. The project is distinguished by its distributed training capabilities, which utilize parameter sharding, mixed-precision training, and CPU offloading to handle models that exceed single-device memory. It also includes specializ
Binarizes raw text datasets into a format optimized for efficient loading during model training and evaluation.
Python
