Awesome GitHub RepositoriesAutomated Data Workflows

Metabase is a business intelligence platform designed to connect to various storage systems and relational databases for data exploration, visualization, and reporting. It provides a centralized environment where users can build queries through a graphical interface or raw code, transforming raw information into interactive dashboards and charts. The platform is built to support self-service analytics, allowing non-technical team members to extract insights without requiring deep knowledge of database syntax. The platform distinguishes itself through a metadata-driven modeling layer that abst

DB-GPT is an AI-driven database management system that uses agentic reasoning to execute data tasks. It converts natural language prompts into executable database queries and combines structured database records with unstructured knowledge bases to provide grounded analysis. The system orchestrates multi-step reasoning chains that integrate database queries, custom scripts, and external tool calls. It allows for the packaging of domain knowledge into reusable analysis skills and executes generated code within sandboxed environments for system safety. The platform covers data orchestration ac

This project is a comprehensive framework for engineering financial data pipelines, designed to automate the collection, cleaning, and synchronization of large-scale market datasets. It functions as a quantitative trading data engine, providing the infrastructure necessary to manage historical and real-time asset pricing information for research and machine learning workflows. The system distinguishes itself through a configuration-driven approach to orchestration, allowing users to manage complex data acquisition tasks across multiple financial providers. It features resilient middleware tha

Pygwalker is a library that transforms tabular data into interactive, drag-and-drop interfaces for exploratory analysis and visualization. It functions as a grammar-based framework that translates user interactions into declarative chart definitions, allowing for the creation of dynamic data exploration environments directly within notebooks or embedded web applications. The system distinguishes itself by offloading heavy analytical computations to backend kernels, which maintains responsiveness when visualizing large datasets. It supports the serialization of visual states into portable conf

SFTPGo is a multi-protocol file server and secure file transfer gateway that enables the exchange and storage of files via SFTP, FTP, HTTP, and WebDAV. It provides a unified management interface and a web-based file manager for configuring server settings, user permissions, and collaborating on documents. The project functions as a cloud storage gateway, bridging legacy file transfer protocols to modern backends such as S3, Google Cloud Storage, and Azure Blob. It features an automated data workflow engine that uses dynamic logic and virtual folders to route files and manage data lifecycles a

DataHub is a metadata management platform designed to unify technical, operational, and business context across diverse data ecosystems. By utilizing a graph-based metadata model and an event-driven ingestion architecture, it creates a centralized source of truth that maps complex data relationships, lineage, and ownership. This foundational framework enables organizations to maintain a synchronized view of their data landscape, supporting both human-led discovery and automated data operations. The platform distinguishes itself through its focus on grounding artificial intelligence and autono

Joyagent-jdgenie is an automated data orchestrator designed to centralize the retrieval and processing of information from disparate remote sources. It functions as a framework for building repeatable data pipelines that fetch, clean, and normalize raw input into consistent, structured formats. The system utilizes a schema-driven engine to apply validation rules and structural templates to incoming data, ensuring compatibility across enterprise systems. By employing configuration-based workflow definitions, it allows for the orchestration of modular tasks into automated execution flows, separ

Keystone is a GraphQL headless content management system and Node.js backend framework. It functions as a schema-driven data manager that automatically generates a GraphQL API and a React administrative dashboard based on a central data model configuration. The system uses an adapter-based database abstraction to decouple core logic from storage layers and a storage-provider abstraction for managing media assets via local or cloud drivers. It distinguishes itself through a component-based field rendering system and a hook-based middleware pipeline for data validation and automation. The fram

Rowy is a low-code backend platform for Firebase that provides a NoSQL spreadsheet interface for managing databases and deploying cloud functions. It functions as a cloud function workflow orchestrator and a database management tool that presents collections as editable grids for visual record editing and bulk data manipulation. The platform distinguishes itself by allowing users to trigger custom server-side scripts and business logic based on database changes and external events. It includes a role-based access control system for defining specific permissions for database collections, indiv

هذا المشروع هو لوحة تحكم لنشاط اللياقة البدنية وخط أنابيب بيانات يجمع إحصائيات الجري والمسارات من منصات رياضية خارجية في ملخص بصري شخصي. يتضمن أداة ترحيل بيانات اللياقة البدنية لتوحيد السجلات بين خدمات رياضية مختلفة والتخزين المحلي، إلى جانب مستورد بيانات لمعالجة ملفات GPX و TCX و FIT. يتميز النظام بخط أنابيب مزامنة نشاط تلقائي يجلب بيانات اللياقة البدنية بشكل دوري ويعيد نشر لوحة تحكم ثابتة إلى منصة استضافة. كما يوفر واجهة مستخدم طرفية تعتمد على لوحة المفاتيح لتصفية وعرض إحصائيات الجري وقوائم النشاط محلياً. تغطي المنصة العديد من مجالات القدرة، بما في ذلك تصور بيانات النشاط من خلال مخططات SVG وخرائط تفاعلية، وأرشفة وترحيل بيانات اللياقة البدنية، والمزامنة التلقائية التي يتم تشغيلها بواسطة سير العمل أو اختصارات الهاتف المحمول. تم بناء واجهة الويب باستخدام إنشاء الموقع الثابت مع التوجيه من جانب العميل ورؤوس الأمان لإدارة حركة المرور.

Autopsy is a digital forensic analysis platform and evidence management suite used to process disk images and file systems. It provides a graphical interface for performing deep forensic examinations of computer hard drives to identify and extract digital artifacts for investigations. The platform is built as a Java-based forensic framework that integrates native libraries to perform direct disk image analysis. It utilizes a modular architecture, allowing for the extension of data ingestion and report generation through the use of plugins. The system manages digital evidence within a central

The platform is a distributed system designed for real-time data monitoring, continuous graph-based query processing, and reactive event automation. It functions as a middleware solution that tracks state changes in external databases and systems, evaluating these streams against graph patterns to identify significant events and state transitions without the need for manual polling. The platform distinguishes itself through its ability to synchronize state updates across distributed environments, including real-time updates to vector databases for AI applications. It utilizes a pluggable conn