2 个仓库
Awesome GitHub RepositoriesDistributed Dataset Mergers
Systems for merging large datasets via repartitioning and shuffling.
Distinct from Distributed Datasets: Focuses on shuffle-based merging for large datasets, distinct from general distributed datasets.
Explore 2 awesome GitHub repositories matching data & databases · Distributed Dataset Mergers. Refine with filters or upvote what's useful.
Awesome Distributed Dataset Mergers GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- dask/dask
dask/dask
13,746在 GitHub 上查看Dask 是一个并行计算框架和分布式任务调度器，旨在将 Python 数据科学工作流从单机扩展到大型集群。它作为一个集群资源管理器，通过将任务及其依赖项表示为有向无环图来编排计算逻辑。这种架构允许系统在管理复杂执行要求的同时，自动将工作负载分配到可用硬件上。 该项目通过一个延迟评估引擎脱颖而出，该引擎将数据操作推迟到明确请求时才执行，从而实现全局图优化和高效的资源分配。它结合了内存感知数据溢出功能，以防止在处理超过可用内存的数据集时系统崩溃，并利用任务图融合将操作序列组合成单个执行步骤，从而最大限度地减少调度开销和节点间通信。 该平台为大规模数据分析提供了全面的功能面，包括对分布式机器学习、高性能计算集成和并行数据处理的支持。它提供了用于集群生命周期管理、性能分析和任务执行实时监控的广泛工具。用户可以在各种基础设施上部署这些环境，包括本地硬件、云提供商、容器化系统和高性能计算集群。
Combines large datasets by repartitioning and shuffling data across nodes to align matching keys.
Pythondasknumpypandas - datajuicer/data-juicer
datajuicer/data-juicer
6,574在 GitHub 上查看Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys
Repartitions distributed datasets into a target number of blocks for optimized processing.
Pythondatadata-analysisdata-pipeline
