4 个仓库
Awesome GitHub RepositoriesParallel Dataframe Operations
Capabilities to automatically distribute tabular data computations across all available CPU cores.
Distinct from Dataframe Processing: Focuses specifically on the parallel execution of dataframe operations rather than general programmatic manipulation.
Explore 4 awesome GitHub repositories matching data & databases · Parallel Dataframe Operations. Refine with filters or upvote what's useful.
Awesome Parallel Dataframe Operations GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- modin-project/modin
modin-project/modin
10,389在 GitHub 上查看Modin is a distributed dataframe library and parallel data processing engine designed to handle large datasets that exceed system memory. It functions as a distributed computing framework that parallelizes data manipulation tasks across multiple CPU cores or clusters to increase throughput and avoid memory errors. The project mirrors the Pandas API, allowing for the distribution of data workflows without changing core code logic. It utilizes a pluggable backend interface, which enables users to switch between different distributed execution engines to optimize performance based on available h
Distributes data and computations across all available CPU cores to accelerate processing speeds.
Pythonanalyticsdata-sciencedataframe - nixtla/statsforecast
Nixtla/statsforecast
4,809在 GitHub 上查看statsforecast 是一个高性能统计时间序列预测库，旨在生成点预测和预测区间。它作为一个分布式时间序列框架，利用基于 C 的预测引擎和自动模型选择器来识别并拟合数据集中每个唯一序列的最佳统计模型。该系统还包括一个时间序列异常检测器，通过将观测值与概率预测区间进行比较来识别异常数据点。 该项目的特色在于其处理数百万个独立序列的大规模并行预测的能力。它通过分布式计算框架、多核并行执行和加速核心 ARIMA 及指数平滑逻辑的编译 C 内核来实现这一点。该系统进一步利用长格式数据布局和惰性求值数据流水线来优化大规模处理，以减少内存开销。 该库提供了一套全面的模型，包括 AutoARIMA、用于间歇性或季节性需求的各种指数平滑方法、Theta 分解以及用于金融风险的 GARCH 波动率建模。它涵盖了更广泛的功能领域，例如带有外生变量的多元预测、时间序列分解以及通过历史交叉验证和滑动窗口分析进行模型评估。 该库与 Polars 等高性能数据结构集成，并提供将保存的模型作为 REST 端点提供服务以进行网络可访问预测的实用程序。
Automatically distributes tabular data computations across all available CPU cores for high-speed processing.
Python - rdatatable/data.table
Rdatatable/data.table
3,894在 GitHub 上查看该项目是一个针对 R 的高性能表格数据处理框架，旨在以内存效率和速度处理海量数据集。它提供了一种增强的数据结构，利用引用语义和就地修改来执行复杂的转换，而无需不必要的对象复制开销。 该库凭借其底层架构优化脱颖而出，包括多线程并行处理、基数排序和内存映射文件解析。通过将关键的数据操作和聚合例程卸载到编译后的 C 代码，它实现了对原本计算昂贵的任务的快速执行。其核心引擎支持高级关系操作，如非等值连接、滚动连接和重叠区间连接，以及用于加速重复数据访问的自动二级索引。 除了主要的处理功能外，该项目还提供了一套全面的数据生命周期管理工具。这包括具有自动类型检测的高速摄取和序列化工具，以及对时间序列分析和多维聚合的专门支持。该框架旨在实现可扩展性，允许用户在包含数十亿行的数据集上执行复杂的分组、过滤和重塑操作，同时保持系统稳定性和性能。
Utilizes multi-threading to speed up computationally intensive data processing tasks across large datasets.
R - nalepae/pandarallel
nalepae/pandarallel
3,802在 GitHub 上查看Pandarallel is a Python library designed to accelerate data manipulation workflows by distributing computationally intensive tasks across all available CPU cores. It functions as a parallel processing utility that integrates directly into standard data analysis pipelines to reduce the execution time of large-scale operations. The library employs a wrapper pattern that intercepts standard data processing calls, redirecting them from sequential execution to a parallelized architecture. It manages this distribution through a multiprocessing-based task model, which utilizes shared-memory serializ
Distributes computationally intensive data transformations across multiple processor cores to significantly reduce execution time for large datasets.
Python
