Awesome GitHub RepositoriesData Stream Integrations

DuckDB is an in-process analytical database engine designed to run directly within an application process. As a zero-dependency, embedded system, it provides enterprise-grade SQL data processing capabilities without the overhead of managing a dedicated database server. It is built to handle complex analytical and aggregation tasks by storing and retrieving information in columns, allowing for high-performance relational data manipulation. The engine distinguishes itself through a columnar vectorized execution model that maximizes CPU cache efficiency during query operations. It employs adapti

This project is a collection of educational resources and reference implementations for the Apache Flink stream processing framework. It provides a learning resource focused on mastering distributed stream processing through implementation guides, performance tuning tutorials, and practical examples. The repository features detailed walkthroughs for building real-time data pipelines using the DataStream and Table APIs. It includes specific integration examples for connecting Apache Flink with Kafka brokers and Elasticsearch indices, as well as reference implementations for real-time deduplica

Open MCT is a web-based framework designed for visualizing telemetry data and monitoring the health of complex systems. It provides a centralized environment for ingesting, processing, and displaying real-time and historical data streams through customizable operator dashboards. The platform is built on a modular architecture that allows for the integration of external data sources and the addition of custom features through a plugin system. By utilizing a hierarchical object-graph model and a unified interface for time-series data, the framework ensures that information is consistently repre

Redpanda is a distributed event streaming engine designed to serve as a high-performance, drop-in replacement for existing event-driven architectures. It provides a foundation for building and scaling applications that require reliable data movement, analytical querying, and strict operational compliance across both cloud and self-managed environments. The platform distinguishes itself through a shared-nothing architecture that utilizes thread-per-core execution and a non-blocking asynchronous input/output engine to maximize throughput. It maintains data consistency through a consensus-based

Orleans is a .NET distributed actor framework designed for building scalable, cloud-native applications. It implements a virtual actor model where entities with stable identities manage their own state and lifecycle across a cluster of servers. The framework provides a distributed state management system with ACID transaction support and a distributed pub/sub streaming engine for real-time data processing. It distinguishes itself through location-transparent routing, automatic actor activation and deactivation, and elastic cluster scaling that redistributes workloads during node failures. Th

Mage AI is a Python-based data pipeline orchestrator and self-hosted data integrated development environment. It is designed for building, scheduling, and monitoring data workflows using a block-based pipeline design and interactive notebook interface. The platform distinguishes itself by integrating generative AI capabilities, allowing users to connect large language model providers via API to incorporate artificial intelligence into automated data streams. It also functions as an Apache Spark data processor, managing the kernels and infrastructure required for high-volume analytics and larg

Horizon is a realtime API server and RethinkDB backend designed to push database changes instantly to front-end clients. It utilizes a WebSocket data streaming API to synchronize data between the database and user interfaces without requiring manual polling. The project integrates an OAuth identity manager for verifying user identities through third-party providers and a role-based access control system to define granular permissions for viewing or modifying database documents. It is delivered as a containerized backend framework, allowing the server and its dependencies to be deployed as a p

X-Ray هو إطار عمل لكشط الويب ومزاحف ويب غير متزامن مصمم لاستخراج البيانات المهيكلة من المواقع. يعمل كمستخرج بيانات HTML يحول محتوى الصفحة الخام إلى مخطط محدد باستخدام محددات بنمط CSS. يطبق المشروع مزاحف متصفح بدون واجهة رسومية قادراً على تنفيذ JavaScript لعرض المحتوى الديناميكي. يتعامل مع اكتشاف محتوى الموقع من خلال استراتيجية زحف بالعرض أولاً واكتشاف الترقيم التلقائي لاجتياز مجموعات النتائج متعددة الصفحات. يدير إطار العمل خطوط أنابيب بيانات الويب باستخدام قائمة انتظار طلبات محدودة التزامن والتحكم في معدل الطلبات لتنظيم مكالمات الشبكة الصادرة. تتم معالجة النتائج المستخرجة عبر استمرارية البيانات القائمة على التدفق لمعالجة مجموعات البيانات الكبيرة دون تحميل ذاكرة النظام بشكل زائد.

Inngest is a durable execution framework and event-driven automation engine designed to orchestrate background workflows. It enables developers to build resilient, stateful processes by memoizing function steps, ensuring that long-running tasks can automatically resume from the last successful operation after failures, timeouts, or infrastructure restarts. The platform distinguishes itself through its event-driven architecture, which uses a schema-validated bus to trigger functions and coordinate complex, multi-step logic. It employs an onion-model middleware approach for cross-cutting concer

dlt هي أداة لاستيعاب البيانات بلغة Python وإطار عمل لخط أنابيب ETL مصمم لجلب البيانات من مصادر متنوعة وحفظها في وجهات مهيكلة. تعمل كمحرك لاستنتاج المخطط (schema inference) يكتشف تلقائياً أنواع البيانات ويسطح هياكل JSON المتداخلة في جداول علائقية، ناقلاً البيانات من المصادر إلى بحيرات البيانات، أو المستودعات، أو قواعد بيانات المتجهات. يتميز المشروع بتوليد خط أنابيب مدعوم بالذكاء الاصطناعي، باستخدام نماذج لغات كبيرة لسقالات كود الاستخراج والموصلات لـ REST APIs. كما يدعم تخزين المتجهات متعدد الوسائط والتعبئة المتخصصة لقواعد بيانات المتجهات لدعم تطبيقات الذكاء الاصطناعي والتعلم الآلي. يغطي إطار العمل مجموعة واسعة من القدرات بما في ذلك تطور المخطط المؤتمت، وتحميل البيانات التزايدي عبر تتبع الحالة، والتحقق من جودة البيانات من خلال فرض عقود البيانات. يوفر أدوات لتطبيع البيانات العلائقية، وتحويلات ما قبل وما بعد التحميل، ومجموعة متنوعة من محولات الوجهة لقواعد بيانات SQL ومخازن الكائنات السحابية. تتم إدارة المراقبة من خلال لوحات معلومات تنفيذ خط الأنابيب، وتتبع نسب الأعمدة، والتحقق من إصدار المخطط باستخدام التجزئات القائمة على المحتوى.

BigchainDB is a blockchain data management system designed to store large datasets on a distributed ledger while maintaining the query performance of a traditional database. It provides immutable record storage for data and digital assets, ensuring a verifiable history of all entries. The project integrates a decentralized blockchain structure with a NoSQL database backing to enable efficient indexing and complex data lookups. It uses a consensus-based state replication model and immutable transaction logging to prevent unauthorized record alteration. The system exposes the ledger state and