1 repositorio
Awesome GitHub RepositoriesSample Splitters
Operators that split batched samples back into their constituent individual samples.
Distinct from Batch Aggregators: Distinct from Batch Aggregators: focuses on splitting batches into individuals rather than combining data points into summaries.
Explore 1 awesome GitHub repository matching data & databases · Sample Splitters. Refine with filters or upvote what's useful.
Awesome Sample Splitters GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- datajuicer/data-juicer
datajuicer/data-juicer
6,574Ver en GitHubData-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys
Provides operators to split batched samples back into individual samples.
Pythondatadata-analysisdata-pipeline
