7 个仓库
Orchestrating large-scale data processing tasks across a cluster of compute nodes.
Distinct from Cloud Batch Processing: Focuses on SLURM-driven batch processing for datasets, not cloud-native training trials.
Explore 7 awesome GitHub repositories matching data & databases · Cluster Batch Processing. Refine with filters or upvote what's useful.
Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis
Processes finite datasets to perform repetitive tasks like report generation.
Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer
Launches a standalone job that builds segments from source data and pushes them to the cluster.
This is an educational repository that teaches container orchestration fundamentals through hands-on guides and practical examples. It covers core Kubernetes concepts including Pods, Services, Deployments, and Namespaces, providing step-by-step exercises that demonstrate how to manage containerized applications across a cluster. The tutorials walk through essential Kubernetes capabilities such as decoupling configuration from application code using ConfigMaps and Secrets, exposing Pods with stable network endpoints via Services, and routing external traffic with Ingress controllers. The mater
Covers running batch jobs to completion using Kubernetes Job resources.
该项目是 Node.js 后端应用测试的最佳实践综合指南和集合。它提供了一套精选的模式和参考示例,用于编写可靠的单元测试、集成测试和组件测试。 该项目通过特定的后端集成策略脱颖而出,包括针对 OpenAPI 规范和共享模式进行 API 合约测试的详细方法。它提供了关于管理消息队列测试的专业指导,重点关注幂等性、弹性和异步事件同步。 该指南涵盖了广泛的能力领域,包括数据库状态隔离和清理、通过网络拦截器和类型安全存根进行的外部依赖模拟,以及容器化测试基础设施的自动化。它还通过使用 RAM 磁盘和内存引擎进行数据存储,解决了测试套件的性能优化问题。
Provides methods to test message queue consumers using batches with mixed failures to ensure system resilience.
This project is a Python software development kit and framework for building applications that integrate with large language models. It serves as a multimodal content generator and vector embedding library, enabling the production and editing of text, images, audio, and video. The toolkit provides specialized capabilities for adapting base models through supervised and reinforcement training. It further distinguishes itself by offering tools for orchestrating complex workflows, including stateful chat sessions, the enforcement of structured output via schemas, and the integration of external
Implements batch processing jobs for handling large volumes of requests with configurable dataset inputs and outputs.
Good Job is a background job processor for Ruby on Rails that utilizes a PostgreSQL database as its primary storage engine. By leveraging relational database transactions, it ensures persistent and reliable task execution, integrating directly with the Active Job framework to handle asynchronous operations and recurring job scheduling within existing application environments. The system distinguishes itself through an in-process execution model that allows background workers to run within the same process as the web server, simplifying deployment by removing the need for separate worker servi
Allows grouping multiple tasks into a single collection to track collective progress and trigger lifecycle callbacks.
Map-anything is a 3D scene reconstruction framework and neural geometry estimator designed to transform two-dimensional images into metric three-dimensional spatial representations using feed-forward neural networks. It provides a specialized toolkit for predicting camera intrinsics and ray directions from single images without requiring external geometric metadata. The project includes a 3D model benchmarking suite that utilizes a unified model wrapper to standardize outputs from diverse reconstruction models. This allows for consistent evaluation and accuracy measurement across various spat
Automates the execution of data processing stages across multiple datasets using a SLURM cluster launcher.