grosser/parallel
Parallel
Parallel is a Ruby library and multi-process execution framework designed to accelerate CPU-intensive operations. It functions as a parallel job orchestrator and concurrent task runner that enables the execution of code across multiple processes or threads.
The project distinguishes itself through secure inter-process communication, utilizing signed data serialization to prevent the injection of forged payloads between parent and child processes. It further differentiates its worker management by assigning unique identifiers to individual processes to prevent collisions when accessing shared external resources.
The library covers broad capability areas including lifecycle management for terminating active tasks, dynamic job generation for distributing workloads, and resource allocation based on hardware processor capacity. It also provides observability tools for real-time monitoring, such as progress bars and main-thread callbacks that track task completion.
Features
- Process-Based Parallelism - Spawns multiple independent system processes to bypass the Ruby Global Interpreter Lock for CPU-intensive tasks.
- Parallel Job Orchestrators - Functions as a parallel job orchestrator coordinating distributed workloads with integrated progress tracking and resource management.
- Signed Serializers - Implements customizable serializers that use verification signatures to safely transfer data between isolated processes.
- Signed Payloads - Protects inter-process communication by signing serialized data to prevent the injection of forged payloads.
- Concurrent Task Execution - Allows blocks of code to be executed across multiple processes or threads for parallel computation.
- Dynamic Job Managers - Processes items from a queue or function to create and distribute workloads during runtime execution.
- Concurrent Thread Execution - Utilizes lightweight execution threads within a single process for I/O-bound tasks or shared-memory operations.
- Parallel Processing Libraries - Provides a specialized library for executing Ruby code concurrently across multiple processes or threads to speed up CPU-intensive operations.
- Ruby Parallel Processing - Executes CPU intensive tasks across multiple processes or threads to reduce total runtime in Ruby applications.
- Inter-Process Communication Security - Uses signed serialization to securely transfer data and commands between parent and child processes.
- Concurrent Task Runners - Provides a system for processing collections of items in parallel with support for dynamic job generation and flow control.
- Dynamic Job Distribution - Provides the ability to distribute workloads to worker processes on the fly using queues or generator functions.
- Inter-Process Message Passing - Implements mechanisms for exchanging structured data and task results between a parent process and its spawned children.
- Inter-Process Communication Management - Provides comprehensive orchestration of the lifecycle and secure data exchange between a main process and its subprocesses.
- Parallel Task Execution - Processes collections of items concurrently using multiple execution units to accelerate CPU-intensive operations.
- Process Coordinators - Manages the lifecycle and resource allocation of multiple child processes to perform parallel operations safely.
- Concurrent Data Processing - Processes large collections of items simultaneously using multiple workers to speed up heavy computational workloads.
- Parallel Execution Termination - Provides mechanisms to halt active processing by completing current items or immediately terminating all worker processes.
- Worker Resource Allocation - Provides configuration settings to define the number of concurrent workers based on available hardware processors.
- Worker Identity Assignments - Assigns unique identifiers to workers to prevent collisions when accessing shared external resources.
- Execution Flow Control - Allows the termination of active tasks using kill signals or return values to exit parallel loops early.
- Main-Thread Progress Coordination - Synchronizes worker completion events with the main thread to trigger callbacks and progress bar updates.
- Worker-Local Storage Mapping - Assigns unique identifiers to each process to prevent collisions when accessing shared external resources.
- Task Progress Monitors - Provides utilities for tracking the real-time progress and status of active parallel tasks.
- Job Execution Tracking - Monitors the lifecycle and progress of computational tasks through real-time item indices and results.
- Real Time Process Monitors - Tracks the progress of parallel jobs using callbacks and progress bars to monitor execution status.
- Concurrency and Parallelism - Facilitates running code in parallel across multiple CPUs.
- Pipeline and Task Automation - Supervisor for parallel execution across CPUs or threads.
- Ruby Frameworks and Tools - Parallel processing library for Ruby.
