3 Repos
Awesome GitHub RepositoriesConcurrent Data Processing
Simultaneous processing of large datasets across multiple workers to improve computational speed.
Distinct from Concurrent Stream Processing: General-purpose concurrent processing of item collections, not limited to streams or network packets.
Explore 3 awesome GitHub repositories matching data & databases · Concurrent Data Processing. Refine with filters or upvote what's useful.
Awesome Concurrent Data Processing GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- grosser/parallel
grosser/parallel
4,261Auf GitHub ansehenParallel is a Ruby library and multi-process execution framework designed to accelerate CPU-intensive operations. It functions as a parallel job orchestrator and concurrent task runner that enables the execution of code across multiple processes or threads. The project distinguishes itself through secure inter-process communication, utilizing signed data serialization to prevent the injection of forged payloads between parent and child processes. It further differentiates its worker management by assigning unique identifiers to individual processes to prevent collisions when accessing shared
Processes large collections of items simultaneously using multiple workers to speed up heavy computational workloads.
Ruby - more-itertools/more-itertools
more-itertools/more-itertools
4,074Auf GitHub ansehenmore-itertools ist eine Erweiterungsbibliothek für das Python-Modul itertools. Sie dient als Toolkit zur Manipulation von Iterables und bietet eine breite Palette an Routinen für Datentransformation, kombinatorische Generierung und Iterator-Zustandsverwaltung. Die Bibliothek zeichnet sich durch fortgeschrittene Zustandsverwaltung und komplexe Sequenzgenerierung aus. Sie bietet Funktionen zum Einsehen zukünftiger Elemente, zum Suchen innerhalb von Sequenzen und zum Erzeugen einzigartiger Permutationen, Kombinationen und Mengenpartitionen aus Sammlungen, die doppelte Elemente enthalten können. Der Funktionsumfang deckt Datenverarbeitungsaufgaben wie rekursives Flattening, Gruppierung, Padding und Umformung von Datenströmen ab. Sie enthält zudem Dienstprogramme für Stream-Merging, Windowing für lokale Nachbarschaftsanalysen und thread-sichere Iterationssynchronisation. Das Projekt bietet zudem spezialisierte Routinen für die Verarbeitung numerischer Sequenzen, einschließlich Matrixmultiplikation, diskreter linearer Faltung und Fourier-Transformationen.
Supports concurrent processing of multiple data collections to improve throughput for large datasets.
Python - gildas-lormeau/zip.js
gildas-lormeau/zip.js
3,859Auf GitHub ansehenzip.js is a JavaScript library designed for creating, reading, and extracting ZIP archives directly within a web browser. It provides a comprehensive toolkit for managing compressed files and encrypted data storage entirely on the client side, eliminating the need for server-side backends or external dependencies. The library distinguishes itself by utilizing the Web Streams API and multi-core processing to handle large datasets efficiently. By offloading heavy compression and decompression tasks to background worker threads, it ensures that the browser interface remains responsive during int
Utilizes background worker pools and asynchronous operations to execute multiple read and write tasks simultaneously, maintaining application responsiveness during heavy processing.
JavaScriptaes-encryptionbrowsercompression-stream
