Awesome GitHub RepositoriesData Processing Pipelines

This project is an algorithmic trading platform designed to automate financial market analysis and the execution of investment strategies. It provides an end-to-end environment for processing real-time market data through automated decision models, allowing for the triggering of financial transactions based on predefined quantitative signals and risk parameters without manual intervention. The platform distinguishes itself through a modular pipeline architecture that decouples data ingestion, signal generation, and trade execution, facilitating the iterative refinement of investment models. I

Nushell is a cross-platform shell and programming language designed to treat all input and output as structured data rather than raw text streams. By enforcing data types and command signatures, it provides a consistent environment for building robust, pipeline-oriented workflows. The shell allows users to chain commands that pass structured objects between stages, enabling complex data processing and automation tasks that remain predictable across different operating systems. What distinguishes the project is its focus on interactive data exploration and modular extensibility. Users can quer

MediaPipe is a cross-platform machine learning framework designed for deploying vision, audio, and text processing models across mobile, desktop, and web environments. It functions as an on-device inference engine that executes complex models locally on edge hardware, ensuring low latency and privacy without requiring a constant internet connection. The framework utilizes a graph-based pipeline orchestration system where data flows through a directed network of modular calculators to ensure synchronized and deterministic processing. It distinguishes itself through a unified runtime that provi

Web-check is a self-hosted diagnostic platform designed to perform comprehensive technical reconnaissance and security audits on web domains. It functions as a network scanner that inspects infrastructure by querying IP addresses, DNS records, SSL certificate chains, and server headers to identify potential misconfigurations or vulnerabilities. The platform is built to run within private infrastructure, ensuring that site investigations remain independent of external tracking or third-party data logging. By utilizing server-side request proxying, the tool bypasses client-side security restric

ArchiveBox is a self-hosted archiving tool designed for personal digital preservation and research data management. It functions as an automated web preservation engine that monitors URL inputs from bookmarks, browser history, or manual entries to capture and store permanent, offline copies of web content. By utilizing headless browser automation, the system renders dynamic web pages to ensure that captured snapshots, PDFs, and media assets remain accurate and accessible even if the original source disappears. The project distinguishes itself through a modular extractor pipeline and a task-qu

SkyWalking is a comprehensive observability stack and application performance monitoring platform. It functions as a distributed tracing system and an AI application monitor, providing a centralized suite for collecting and analyzing logs, metrics, and traces to maintain the health of containerized architectures. The platform distinguishes itself through a service topology visualizer that renders interactive maps of infrastructure dependencies and communication patterns. It also includes specialized capabilities for generative AI workflow observation to track the execution flow and performanc

This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, whi

Telegraf is a modular, cross-platform telemetry pipeline designed to collect, process, and route metrics from diverse infrastructure, applications, and hardware. It functions as a server-side middleware that normalizes heterogeneous data into a unified format, enabling consistent monitoring across complex environments. By utilizing a plugin-driven architecture, the agent manages the entire lifecycle of telemetry data from initial ingestion to final transmission. The project distinguishes itself through a declarative, configuration-driven execution model that allows users to define complex dat

This application is a cross-platform desktop utility designed for automated translation, optical character recognition, and speech synthesis. It functions as a modular client that integrates various local and remote language services, allowing users to process text through hotkeys, clipboard monitoring, or direct input. The software distinguishes itself through a plugin-based architecture and a built-in automation framework. By exposing a local network interface, it enables external applications and scripts to programmatically trigger its translation and recognition workflows. Users can furth

Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

OpenProject ist eine Open-Source-Plattform für Arbeitsmanagement und Projektportfolios, die für die Verfolgung von Aufgaben, die Verwaltung von Projektlebenszyklen und die Überwachung strategischer Ziele entwickelt wurde. Sie bietet eine zentralisierte Umgebung zur Organisation von Team-Workflows durch agile Boards, Gantt-Diagramme und Roadmaps und unterstützt gleichzeitig komplexe Projekthierarchien sowie Ressourcenkapazitätsplanung in großen Organisationen. Die Plattform zeichnet sich durch spezialisierte Unterstützung für Building Information Modeling (BIM) aus, die es Teams ermöglicht, 3D-Modelle und Konstruktionsprobleme direkt in einem Webbrowser zu importieren, zu visualisieren und zu koordinieren. Sie differenziert ihre Fähigkeiten weiter durch die Integration von Zustandsautomaten-Workflow-Engines und relationaler Arbeitspaket-Modellierung, die eine hochgradig angepasste, schema-gesteuerte Verfolgung von Projektelementen, Abhängigkeiten und Statusübergängen ermöglichen. Das System deckt ein breites Funktionsspektrum ab, einschließlich Identitätsmanagement auf Unternehmensebene, automatisierter Berichterstattung und Überwachung der finanziellen Leistung. Es bietet umfangreiche Konfigurationsoptionen für benutzerdefinierte Arbeitstypen, Projekt-Governance-Rollen und skalierte agile Methoden sowie programmatische Schnittstellen für die Automatisierung externer Systeme und die Anbindung von KI-Assistenten. Die Software ist für die selbst gehostete Bereitstellung konzipiert und bietet umfassende Verfahren für Installation, Wartung und Datenmigration von Drittsystemen.

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

Cocoindex is an incremental data processing engine that builds and maintains live indexes for AI agents, with a core focus on codebase indexing and knowledge graph extraction. The engine uses a function-graph execution model where user-defined Python functions are composed into a directed acyclic graph, and it processes data incrementally so only changed source records or code paths are re-computed, avoiding full recomputation at any scale. It supports automatic schema inference from transformation pipeline type annotations and provides full data lineage tracing, tagging every output record wi

Apache Hudi is an open-source table format that brings ACID transactions, incremental processing, and multi-modal indexing to data lakes. It provides atomic commits with snapshot isolation, rollback, and optimistic concurrency control for reliable data lake operations, while supporting upserts, record-level updates, and deletions in large analytical datasets. The project distinguishes itself through a timeline-based architecture that coordinates all write operations, enabling features like time-travel querying, incremental change streaming, and multi-modal query views that include snapshot, i

Dieses Projekt ist eine umfassende Bildungsressource und Tutorial-Serie zum Erlernen von Bash-Scripting und Shell-Programmierung. Es dient als technischer Leitfaden und Referenzhandbuch für das Schreiben von Automatisierungsskripten und die Erstellung wiederverwendbarer Werkzeuge in einer Linux-Umgebung. Das Material konzentriert sich auf DevOps- und SysOps-Automatisierung und deckt die Entwicklung benutzerdefinierter CLI-Tools, die Automatisierung der Systemadministration sowie die Implementierung von Sicherheitsautomatisierung wie die Generierung zufälliger Passwörter und DDoS-Schutz ab. Der Umfang umfasst grundlegende Sprachkonzepte wie Variablen, Schleifen und bedingte Logik sowie operative Funktionen wie System-Health-Monitoring, Zusammenfassung von Webserver-Logs und Remote-Server-Management via SSH. Es behandelt zudem die Terminal-Interaktion durch den Aufbau interaktiver Menüs, die Formatierung von Ausgaben und das Aliasing von Befehlen. Der Inhalt ist als eine Reihe von Leitfäden, einem E-Book und einer Skripting-Referenz strukturiert.

Jitsu is an open-source customer data platform designed to orchestrate event data pipelines. It captures, transforms, and routes behavioral data from web and server sources into data warehouses and analytics tools, providing a unified infrastructure for managing event streams. The platform distinguishes itself through its focus on self-hosted, containerized operations that grant users full control over their data security and privacy. It features a robust identity resolution engine that stitches disparate user identifiers into persistent profiles across sessions and devices, alongside program

Grid.js ist eine framework-agnostische JavaScript-Bibliothek für das Rendern interaktiver Datengitter. Sie ermöglicht die Anzeige strukturierter Informationen in tabellarischen Formaten über verschiedene Frontend-Umgebungen hinweg und unterstützt die Datenbefüllung aus statischen Arrays oder JSON-Importen. Die Bibliothek verfügt über ein Plugin-System zur Erweiterung von Benutzeroberflächenkomponenten und Logik sowie über eine benutzerdefinierte Datenpipeline zur Transformation von Informationen vor der Anzeige. Sie enthält integrierte Unterstützung für mehrsprachige Lokalisierung, um Schnittstellenelemente und Nachrichten zu übersetzen. Das Projekt deckt grundlegende Datenvisualisierungsfunktionen ab, einschließlich Sortierung, Filterung und dynamischer Gittervisualisierung. Es bietet Tools zur Verwaltung von Konfigurationseinstellungen und zur Integration in verschiedene JavaScript-Umgebungen.

Dieses Projekt ist eine Echtzeit-Produktempfehlungs-Engine, die auf Apache Flink basiert. Sie fungiert als Streaming-Pipeline für Verhaltensanalysen, die Rohprotokolle verarbeitet, um Benutzerinteressen und Trends zur Produktpopularität abzuleiten. Das System nutzt eine Collaborative-Filtering-Engine, um die Artikelähnlichkeit mittels Kosinus-Ähnlichkeit und gemeinsamer Benutzerinteraktionsmuster zu berechnen. Es verwendet eine hybride Re-Ranking-Pipeline, die globale Popularitätslisten mit personalisierten Benutzerprofilen kombiniert, um empfohlene Artikel zu sortieren. Die Architektur umfasst einen Wide-Column-Benutzerspeicher unter Verwendung von HBase für dauerhafte Verhaltensaufzeichnungen und einen Redis-basierten Cache für Echtzeit-Artikel-Heatlists. Die Pipeline bietet Funktionen für die Extraktion von Verhaltensinteressen, die Analyse von Interaktionsintervallen und ein Performance-Dashboard zur Überwachung der Log-Ingestion-Raten.

DotnetSpider is a .NET web crawling framework and C# data extraction tool designed for automated web page discovery and the retrieval of structured data from the internet at scale. It functions as a high-level web scraping library for collecting information from various websites. The framework provides capabilities for automated web crawling and large-scale data scraping. It enables web content extraction to facilitate the creation of local databases or the analysis of online information through programmatic web automation within the .NET ecosystem. The system utilizes a pipeline-based data