1 个仓库
Awesome GitHub RepositoriesSample Splitters
Operators that split batched samples back into their constituent individual samples.
Distinct from Batch Aggregators: Distinct from Batch Aggregators: focuses on splitting batches into individuals rather than combining data points into summaries.
Explore 1 awesome GitHub repository matching data & databases · Sample Splitters. Refine with filters or upvote what's useful.
Awesome Sample Splitters GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- datajuicer/data-juicer
datajuicer/data-juicer
6,574在 GitHub 上查看Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys
Provides operators to split batched samples back into individual samples.
Pythondatadata-analysisdata-pipeline
