2 个仓库
Awesome GitHub RepositoriesMemory-Aware
Caching systems that store intermediate results while monitoring memory usage to prevent system crashes.
Distinct from Result Caching: Distinct from Result Caching: focuses on memory-aware eviction and capacity management for large-scale data workflows.
Explore 2 awesome GitHub repositories matching data & databases · Memory-Aware. Refine with filters or upvote what's useful.
Awesome Memory-Aware GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- prefecthq/prefect
PrefectHQ/prefect
21,640在 GitHub 上查看Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep
Clears task or flow results from active memory after commitment to storage to prevent system crashes during large-scale data processing.
Pythonautomationdatadata-engineering - dask/dask
dask/dask
13,746在 GitHub 上查看Dask 是一个并行计算框架和分布式任务调度器，旨在将 Python 数据科学工作流从单机扩展到大型集群。它作为一个集群资源管理器，通过将任务及其依赖项表示为有向无环图来编排计算逻辑。这种架构允许系统在管理复杂执行要求的同时，自动将工作负载分配到可用硬件上。 该项目通过一个延迟评估引擎脱颖而出，该引擎将数据操作推迟到明确请求时才执行，从而实现全局图优化和高效的资源分配。它结合了内存感知数据溢出功能，以防止在处理超过可用内存的数据集时系统崩溃，并利用任务图融合将操作序列组合成单个执行步骤，从而最大限度地减少调度开销和节点间通信。 该平台为大规模数据分析提供了全面的功能面，包括对分布式机器学习、高性能计算集成和并行数据处理的支持。它提供了用于集群生命周期管理、性能分析和任务执行实时监控的广泛工具。用户可以在各种基础设施上部署这些环境，包括本地硬件、云提供商、容器化系统和高性能计算集群。
Stores frequently accessed task results in memory to accelerate operations while automatically evicting data to manage capacity.
Pythondasknumpypandas
