Awesome GitHub RepositoriesParallel Processing Pipelines

FFmpeg is a cross-platform multimedia framework designed for the recording, conversion, and streaming of audio and video content. It functions as a comprehensive toolkit that provides both a command-line utility for direct media manipulation and a collection of low-level libraries for integration into custom applications. At its core, the project utilizes a packet-based stream engine and a format-agnostic abstraction layer to handle diverse media standards, containers, and network protocols. The framework distinguishes itself through a modular, graph-based filter execution model that allows f

Langextract is a framework designed to transform unstructured text into structured, machine-readable data using language model orchestration. It provides a high-performance pipeline that processes large volumes of narrative text by utilizing parallel execution and sequential extraction passes. The library is built to handle complex data extraction tasks, including specialized support for clinical information and medical entity relationship recognition. The project distinguishes itself through a plugin-based architecture that supports both local hardware execution and cloud-hosted model endpoi

Kotaemon is an orchestration framework designed for building modular, agentic workflows that integrate document processing, retrieval-augmented generation, and multi-step reasoning. It provides a comprehensive platform for developing document-based question answering systems, allowing users to chain language models, prompt templates, and external tools into complex, automated pipelines. The system distinguishes itself through a highly modular architecture that emphasizes component-based composition and schema-driven data exchange. It supports autonomous agents capable of decomposing complex q

This project is a unified toolchain for JavaScript and TypeScript development, providing a suite of specialized tools for bundling, parsing, linting, and formatting source code. The toolchain is implemented in Rust and utilizes a shared internal representation to allow multiple tools to operate on the same parsed source code. It employs a parallelized analysis pipeline and single-pass parsing with error recovery to process files across CPU cores. The suite covers several core capability areas, including static analysis for identifying programmatic errors, automated style standardization, and

Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing

This project is a structured educational program and comprehensive training curriculum designed to teach the end-to-end lifecycle of machine learning models. It serves as a resource for engineers to master the transition of data science projects from development into reliable, production-ready systems. The curriculum focuses on the practical application of engineering best practices, emphasizing the orchestration of complex data processing and training sequences. It provides instruction on building repeatable workflows, managing experiment metadata, and implementing infrastructure automation

Compromise is a natural language processing library and rule-based text parser designed to analyze unstructured text. It functions as a toolkit for identifying parts of speech, linguistic patterns, and semantic meaning, while providing specialized engines for named entity recognition and the parsing of temporal and numeric data. The project is distinguished by its linguistic morphological engine, which can conjugate verbs across different tenses and inflect nouns and adjectives. It further allows for linguistic model customization through a plugin system that enables the extension of lexicons

PyOD is a Python anomaly detection library used to identify outliers in tabular, time series, graph, text, and image data. It provides a collection of algorithms for detecting anomalous data points and includes a unified detector interface that standardizes input and output signatures across its available detection algorithms. The project features a multi-modal outlier detector for identifying anomalies across diverse formats including unstructured text and images, as well as a specialized toolkit for graph-based and time-series anomaly detection. It includes an ensemble framework for combini

Storm is a distributed stream processing framework and fault-tolerant compute engine designed for executing real-time continuous computations across a cluster of machines. It functions as a stateful stream processor and cluster topology manager, enabling the deployment and monitoring of distributed data flow configurations. The system ensures exactly-once semantics by utilizing transactional state management to guarantee that every message in a data stream is processed exactly one time. It further operates as a distributed RPC system, allowing for the integration of non-native languages throu

Mage AI ist ein Python-basierter Daten-Pipeline-Orchestrator und eine selbstgehostete integrierte Entwicklungsumgebung (IDE) für Daten. Er ist darauf ausgelegt, Daten-Workflows mittels eines blockbasierten Pipeline-Designs und einer interaktiven Notebook-Schnittstelle zu erstellen, zu planen und zu überwachen. Die Plattform zeichnet sich durch die Integration generativer KI-Funktionen aus, die es Benutzern ermöglichen, Anbieter großer Sprachmodelle (LLMs) per API anzubinden, um künstliche Intelligenz in automatisierte Datenströme einzubinden. Sie fungiert zudem als Apache Spark-Datenprozessor und verwaltet die Kernels und die Infrastruktur, die für hochvolumige Analysen und groß angelegte Datenverarbeitung erforderlich sind. Das System deckt ein breites Spektrum an Data-Engineering-Funktionen ab, einschließlich ETL-Workflow-Automatisierung, dbt-Modellverwaltung und Datenstrom-Discovery. Es bietet Werkzeuge für die Versionskontrollintegration via Git, containerisierte Bereitstellung und rollenbasierte Zugriffskontrolle zur Verwaltung von Pipelines in Entwicklungs- und Produktionsumgebungen. Die Überwachung erfolgt durch System-Performance-Telemetrie und Pipeline-Ausführungs-Debugging.

Swarms ist ein Multi-Agenten-Orchestrierungs-Framework und ein Toolkit für autonome Agenten, das darauf ausgelegt ist, Agenten auf Basis von Large Language Models zu koordinieren. Es dient als Workflow-Engine zur Verwaltung von Agentenbeziehungen und bietet die Infrastruktur zum Aufbau autonomer Agenten mit integriertem Gedächtnis, Tool-Calling-Fähigkeiten und Reasoning-Schleifen. Das Framework zeichnet sich durch seine Multi-Agenten-Konsenssysteme aus, die Abstimmungen, gegnerische Debatten und Richter-Agenten nutzen, um qualitativ hochwertige Antworten zu synthetisieren. Es unterstützt eine Vielzahl von Kollaborationsmustern, einschließlich Direktor-Arbeiter-Hierarchien, Experten-Synthese und automatisierter Swarm-Architektur-Generierung basierend auf Beschreibungen in natürlicher Sprache. Das System deckt ein breites Spektrum operativer Fähigkeiten ab, einschließlich Graph-basierter und sequenzieller Workflow-Orchestrierung über eine domänenspezifische Sprache, einheitlicher Schnittstellen für diverse Modellanbieter und Integration mit dem Model Context Protocol für dynamische Tool-Entdeckung. Es enthält zudem Unterstützung für Retrieval Augmented Generation, zustandsbehaftetes Gedächtnis und die Fähigkeit, Agentenfunktionalität als Webdienste bereitzustellen. Das Projekt bietet eine Kommandozeilenschnittstelle für das Agentenmanagement und unterstützt die Konfiguration über YAML- und modulare Markdown-Skill-Dateien.

ClearML is a comprehensive MLOps platform designed to manage the entire machine learning lifecycle. It functions as an experiment tracking tool, a data versioning system, and a pipeline orchestrator, while providing infrastructure for GPU cluster management and model serving. The platform is distinguished by its ability to handle hybrid-cloud compute scheduling and fractional GPU allocation, allowing multiple workloads to share a single hardware accelerator. It employs a metadata-based approach to data versioning, using virtual views to track large datasets and artifacts without duplicating r

Storm is a distributed stream processing framework designed to execute unbounded computations across a cluster to process real-time data streams. It functions as a data pipeline orchestrator that allows users to define and deploy declarative data flow graphs connecting streaming sources to processing components. The system operates as a multi-tenant distributed compute engine that isolates workloads and limits resource usage across shared clusters using dedicated pools and access control. It is also a secure distributed processing engine that employs encrypted node communication and SSL-secur

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

Rushstack ist ein umfassendes Toolset für die Verwaltung groß angelegter TypeScript-Monorepos und bietet ein Framework für Build-Pipeline-Automatisierung, Abhängigkeitskoordination und statische Analyse. Es fungiert als inkrementeller Build-Orchestrator und Managementsystem, das darauf ausgelegt ist, Konsistenz und Performance über mehrere Pakete in einem geteilten Workspace hinweg zu wahren. Das System zeichnet sich durch ein Ausführungsmodell aus, das auf gerichteten azyklischen Graphen (DAGs) und inhalts-hash-basierter Inkrementalität basiert, was sicherstellt, dass nur betroffene Projekte neu gebaut werden. Es optimiert Entwicklungs-Workflows weiter durch Remote-Build-Artifact-Caching mit Integrationen für S3, Azure und HTTP und nutzt symlink-basiertes Workspace-Management für sofortige Sichtbarkeit lokaler Änderungen. Das Toolset deckt ein breites Spektrum an Funktionen ab, einschließlich automatisierter Release-Workflows, API-Vertragsanalyse und der Durchsetzung von Code-Qualität im Unternehmen durch standardisierte Linting-Profile. Es bietet zudem umfassende Unterstützung für Asset-Bundling, TypeScript-Transpilierung und die Generierung von Typdefinitionen aus JSON-Schemas und statischen Assets. Zusätzliche Aufgabenbereiche umfassen Lokalisierungsmanagement, die Generierung von Lizenzhinweisen und die Orchestrierung von Command-Line-Interfaces mit synchronisierter Prozessausgabe.

Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp

Wireit is a build system orchestrator and npm script task runner that manages complex dependencies between scripts. It functions as a development workflow automator by tracking input files and dependencies to skip redundant executions and optimize total build time. The project features a content-addressable build cache that saves generated files to bypass expensive computations. It includes a file watcher that monitors project changes and automatically triggers only the affected scripts within a dependency graph. The tool covers task orchestration through parallel script execution, backgroun

Pachyderm is a containerized, versioned, and lineage-tracked data pipeline platform that runs natively on Kubernetes. It combines a distributed file system backend with immutable data versioning, so every commit to a data repository creates an auditable snapshot, and every pipeline step executes as an isolated container. The platform is defined by a data-centric pipeline model where pipelines are specified by their input and output data repositories rather than explicit task sequences, and provenance is recorded as a directed acyclic graph of commits linking output data to its input sources an

pgloader is a command-line tool that automates the migration of data and schema from various source databases and file formats into PostgreSQL. It combines schema discovery, parallel data pipelines, and type casting into a single, declarative workflow, using PostgreSQL's COPY protocol for high-throughput bulk loading. The tool distinguishes itself by compiling a dedicated command language into concurrent reader-writer pipelines that handle schema introspection, data transformation, and error-resilient batch processing. It supports migrating entire databases from MySQL, MS SQL, SQLite, and Pos

MSBuild ist eine .NET-Build-Engine und ein Software-Build-Automatisierungssystem, das Quellcode in bereitstellbare Anwendungen transformiert. Es nutzt ein deklaratives Projektsystem basierend auf einem strukturierten XML-Schema, um Build-Elemente, Eigenschaften und Targets zu definieren, die im gesamten .NET-Ökosystem und in Visual Studio verwendet werden. Das System bietet ein anpassbares Build-Task-Framework, das die Implementierung benutzerdefinierter Tasks und Logger ermöglicht, um spezifische Software-Konstruktionsoperationen durchzuführen. Diese Erweiterbarkeit ermöglicht die Erstellung spezialisierter Logik zur Modifikation des Verhaltens des Build-Prozesses. Die Engine verwaltet die Build-Automatisierung durch ein aufgabenbasiertes Ausführungsmodell und eine abhängigkeitsbewusste Workflow-Orchestrierung. Ihre Funktionen decken die .NET-Software-Kompilierung, das Deployment für mehrere Ziel-Frameworks und das inkrementelle Build-Tracking ab. Sie bietet zudem ein Command-Line-Interface für Konfigurations-Overrides und ein Logging-System zur Erfassung des Build-Outputs.