Awesome GitHub RepositoriesData Processing Configurations

CrewAI is a multi-agent orchestration framework designed for building autonomous systems that execute complex, multi-step workflows. It provides a development platform where specialized agents are defined with specific roles, goals, and tool sets to perform tasks collaboratively. By leveraging a declarative workflow engine, the system manages task dependencies, state transitions, and execution logic, allowing for the creation of structured, stateful sequences of operations. The framework distinguishes itself through its hierarchical management capabilities, which utilize manager agents to coo

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

This project is a Python-based framework that functions as a generative AI agent for programmatic data analysis. It enables users to interact with structured data sources through natural language prompts, translating these requests into executable code to perform analysis, data cleaning, and visualization. By maintaining conversational context across multi-turn interactions, the system allows for iterative exploration and the building of complex data narratives. The framework distinguishes itself through a robust semantic layer and secure execution model. It maps raw datasets to descriptive m

This project is a collection of educational resources and reference implementations for the Apache Flink stream processing framework. It provides a learning resource focused on mastering distributed stream processing through implementation guides, performance tuning tutorials, and practical examples. The repository features detailed walkthroughs for building real-time data pipelines using the DataStream and Table APIs. It includes specific integration examples for connecting Apache Flink with Kafka brokers and Elasticsearch indices, as well as reference implementations for real-time deduplica

Logstash is a JVM-based event processor and extract, transform, load system designed for log data processing pipelines. It functions as a plugin-based data ingestor that collects, transforms, and delivers logs and event data from multiple sources to various destinations. The system utilizes a modular architecture of interchangeable input, filter, and output components to handle real-time data ingestion and enterprise log aggregation. Users can extend the pipeline's functionality by developing custom plugins to support unique data sources or specific transformation logic. The platform covers

Unstructured is an enterprise-grade data orchestration engine designed to transform raw, unstructured files into structured, machine-readable formats. It functions as a comprehensive platform for document ingestion, partitioning, and enrichment, specifically engineered to prepare complex data for retrieval-augmented generation and agentic AI workflows. The platform distinguishes itself through its sophisticated document processing strategies, which combine rule-based extraction with vision-language models to handle diverse file layouts, tables, and images. It provides a modular architecture t

Rete is a framework for building interactive, node-based visual interfaces and dataflow programming environments. It provides a core engine that processes directed graphs, allowing developers to define modular logic where nodes represent operations and connections represent the flow of data or control. By decoupling the graph logic from the user interface, the framework enables the creation of custom visual editors that can be integrated into various frontend component libraries. The project distinguishes itself through a highly extensible, signal-driven architecture that supports complex req

SeaTunnel is a distributed data integration engine designed to synchronize structured and unstructured data across diverse sources and sinks. It functions as a multi-engine execution framework that can run data integration tasks across different distributed computing backends to optimize workload performance. The project is distinguished by a visual data pipeline designer for configuring workflows without manual code and a specialized change data capture tool for streaming incremental database updates. It also includes an enrichment pipeline that integrates large language models and embedding

Mage AI ist ein Python-basierter Daten-Pipeline-Orchestrator und eine selbstgehostete integrierte Entwicklungsumgebung (IDE) für Daten. Er ist darauf ausgelegt, Daten-Workflows mittels eines blockbasierten Pipeline-Designs und einer interaktiven Notebook-Schnittstelle zu erstellen, zu planen und zu überwachen. Die Plattform zeichnet sich durch die Integration generativer KI-Funktionen aus, die es Benutzern ermöglichen, Anbieter großer Sprachmodelle (LLMs) per API anzubinden, um künstliche Intelligenz in automatisierte Datenströme einzubinden. Sie fungiert zudem als Apache Spark-Datenprozessor und verwaltet die Kernels und die Infrastruktur, die für hochvolumige Analysen und groß angelegte Datenverarbeitung erforderlich sind. Das System deckt ein breites Spektrum an Data-Engineering-Funktionen ab, einschließlich ETL-Workflow-Automatisierung, dbt-Modellverwaltung und Datenstrom-Discovery. Es bietet Werkzeuge für die Versionskontrollintegration via Git, containerisierte Bereitstellung und rollenbasierte Zugriffskontrolle zur Verwaltung von Pipelines in Entwicklungs- und Produktionsumgebungen. Die Überwachung erfolgt durch System-Performance-Telemetrie und Pipeline-Ausführungs-Debugging.

CloudQuery is a cloud infrastructure ETL tool and multi-cloud data pipeline designed to collect, synchronize, and normalize resource metadata from various cloud providers and SaaS platforms. It functions as a centralized asset inventory manager and security posture manager, extracting configuration and state data into relational databases, data lakes, or data warehouses. The system distinguishes itself by transforming complex, nested cloud API responses into flat relational tables, enabling the use of standard SQL for asset querying and analysis. It employs a modular plugin system for data ex

Cocoindex is an incremental data processing engine that builds and maintains live indexes for AI agents, with a core focus on codebase indexing and knowledge graph extraction. The engine uses a function-graph execution model where user-defined Python functions are composed into a directed acyclic graph, and it processes data incrementally so only changed source records or code paths are re-computed, avoiding full recomputation at any scale. It supports automatic schema inference from transformation pipeline type annotations and provides full data lineage tracing, tagging every output record wi

Apache Hive is a SQL-on-Hadoop data warehouse that enables querying and managing petabytes of data stored in distributed storage such as HDFS and cloud storage services. It provides a familiar SQL interface for batch analytics and reporting, supported by a core set of components including the HiveServer2 Thrift service for remote query execution, the Hive Metastore Service for central metadata management, the Hive ACID Transaction Engine for concurrent read-write operations, and the Hive LLAP Interactive Engine for low-latency analytical processing. The WebHCat REST API offers an HTTP interfac

dlt ist ein Python-Tool zur Datenaufnahme und ein ETL-Pipeline-Framework, das darauf ausgelegt ist, Daten aus verschiedenen Quellen abzurufen und in strukturierten Zielen zu speichern. Es fungiert als Schema-Inferenz-Engine, die automatisch Datentypen erkennt und verschachtelte JSON-Strukturen in relationale Tabellen flacht, wobei Daten von Quellen in Lakehouses, Warehouses oder Vektordatenbanken verschoben werden. Das Projekt zeichnet sich durch KI-gestützte Pipeline-Generierung aus, die Large Language Models nutzt, um Extraktionscode und Konnektoren für REST-APIs zu erstellen. Es unterstützt zudem multimodale Vektorspeicherung und die spezialisierte Befüllung von Vektordatenbanken zur Unterstützung von KI- und Machine-Learning-Anwendungen. Das Framework deckt ein breites Spektrum an Funktionen ab, einschließlich automatisierter Schema-Evolution, inkrementellem Datenladen mittels Statusverfolgung und Datenqualitätsvalidierung durch die Durchsetzung von Datenverträgen. Es bietet Tools für relationale Datennormalisierung, Pre- und Post-Load-Transformationen sowie eine Vielzahl von Ziel-Adaptern für SQL-Datenbanken und Cloud-Objektspeicher. Die Observability wird durch Pipeline-Ausführungs-Dashboards, Spalten-Lineage-Tracking und Schema-Versionsverifizierung mittels inhaltsbasierter Hashes gehandhabt.

Fluvio ist eine verteilte Event-Streaming-Plattform und eine Cloud-native Streaming-Engine, die für das Sammeln, Persistieren und Replizieren von Echtzeit-Datenströmen über einen verteilten Cluster hinweg entwickelt wurde. Sie fungiert als Echtzeit-Datenpipeline für den Aufbau zustandsbehafteter Workflows, die Daten zwischen externen Quellen und Senken aufnehmen, anreichern und exportieren. Die Plattform zeichnet sich durch die Verwendung von WebAssembly zur Ausführung kompilierter Module für In-Line-Datentransformationen und -filterung aus. Dies ermöglicht die Ausführung benutzerdefinierter Geschäftslogik, um Informationen während der Übertragung umzuformen, ohne den Cluster neu starten zu müssen. Das System deckt ein breites Spektrum an Funktionen ab, einschließlich connector-basierter Datenaufnahme aus externen Protokollen, log-strukturierter unveränderlicher Speicherung mit Zero-Copy-IO und horizontaler Clusterskalierung. Es unterstützt die Erstellung komplexer ereignisgesteuerter Pipelines, die zustandsbehaftete Verarbeitung, fensterbasierte Aggregationen und partitionierte Datenverteilung nutzen. Die Engine kann als leichtgewichtiges Binärprogramm auf diversen Systemarchitekturen bereitgestellt werden, einschließlich ARM64-IoT-Geräten für die Datenverarbeitung am Edge.

Jitsu is an open-source customer data platform designed to orchestrate event data pipelines. It captures, transforms, and routes behavioral data from web and server sources into data warehouses and analytics tools, providing a unified infrastructure for managing event streams. The platform distinguishes itself through its focus on self-hosted, containerized operations that grant users full control over their data security and privacy. It features a robust identity resolution engine that stitches disparate user identifiers into persistent profiles across sessions and devices, alongside program

RavenDB is a multi-model NoSQL document database designed for high-performance, ACID-compliant data storage. It persists structured information as schema-flexible JSON documents and utilizes a unit-of-work session pattern to track entity changes and batch modifications into atomic transactions. The platform is built on a distributed architecture that supports horizontal scaling through sharding and ensures high availability via multi-node, master-to-master cluster replication. The database distinguishes itself through a self-optimizing query engine that automatically creates and maintains ind

DevLake is a DevOps data platform and analytics tool designed to orchestrate data pipelines that ingest, transform, and sync metadata from external development tools into a unified database. It functions as a system for collecting and normalizing data from source control, CI/CD pipelines, and issue trackers into a standardized schema to enable consistent software delivery analytics. The platform distinguishes itself by transforming tool-specific data into a common domain model, allowing for the calculation of engineering metrics via SQL. It provides specialized frameworks for measuring DORA m

Connector-X is a high-performance SQL data extraction library and bridge for transferring relational database records into memory-efficient data structures. It functions as a parallel database connector and federated query engine capable of executing and joining queries across multiple remote database connections to aggregate data locally. The project distinguishes itself through a zero-copy approach to data loading, which transfers SQL query results into memory structures without duplicating data. It maximizes throughput by partitioning SQL queries into threads, employing parallel columnar a