# datalinkdc/dinky

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/datalinkdc-dinky).**

3,740 stars · 1,345 forks · Java · Apache-2.0

## Links

- GitHub: https://github.com/DataLinkDC/dinky
- Homepage: http://www.dinky.org.cn
- awesome-repositories: https://awesome-repositories.com/repository/datalinkdc-dinky.md

## Topics

`datalake` `datawarehouse` `flink` `flinkcdc` `flinksql` `olap` `real-time-computing-platform` `sql`

## Description

Dinky is a real-time data platform for developing, deploying, and operating streaming applications based on Apache Flink. It functions as a SQL streaming IDE and a real-time data pipeline orchestrator, providing a web-based environment for writing and verifying queries with integrated logic plan visualization and lineage tracking.

The platform acts as a distributed cluster manager, allowing the registration, monitoring, and administration of multiple processing clusters from a centralized interface. It also serves as a change data capture integration tool, synchronizing real-time database changes into warehouses or lakes with automatic schema evolution.

The system covers a broad range of operational capabilities, including data pipeline observability via lineage analysis, enterprise data governance through multi-tenant resource isolation and role-based access control, and fault tolerance using snapshot-based state recovery. Developer experience is supported through custom function management, query result previewing, and a managed editor for SQL application authoring.

Operational oversight is provided through job health monitoring, real-time alert configurations, and execution plan inspection.

## Tags

### Data & Databases

- [Apache Flink Management Platforms](https://awesome-repositories.com/f/data-databases/apache-flink-management-platforms.md) — Acts as a centralized control plane for managing the full lifecycle of Apache Flink streaming and batch applications.
- [SQL Query Execution](https://awesome-repositories.com/f/data-databases/sql-query-execution.md) — Executes SQL queries across different deployment environments using a standardized interface to process data at scale. ([source](http://www.dinky.org.cn/docs/next/openapi/openapi_overview))
- [Change Data Capture](https://awesome-repositories.com/f/data-databases/change-data-capture.md) — Synchronizes databases in real-time to warehouses or lakes with automatic table creation and schema evolution. ([source](http://www.dinky.org.cn/docs/1.0/get_started/overview))
- [Flink Execution Engines](https://awesome-repositories.com/f/data-databases/data-engineering/distributed-compute-frameworks/distributed-computing-engines/feature-materialization-pipelines/flink-execution-engines.md) — Leverages Apache Flink as a distributed processing engine to execute real-time data pipelines and SQL jobs.
- [Data Lake Orchestrators](https://awesome-repositories.com/f/data-databases/data-integration-synchronization/data-integration/cloud-data-lake-integrations/data-lake-orchestrators.md) — Orchestrates the flow of real-time data from various sources into cloud-native data lakes and warehouses. ([source](https://cdn.jsdelivr.net/gh/datalinkdc/dinky@dev/README.md))
- [Logic Verification](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/stream-processing-systems/data-streaming/real-time-stream-monitors/logic-verification.md) — Provides real-time previewing of table contents and custom functions to verify streaming processing logic. ([source](https://cdn.jsdelivr.net/gh/datalinkdc/dinky@dev/README.md))
- [CDC Synchronization](https://awesome-repositories.com/f/data-databases/database-migrations/cdc-synchronization.md) — Uses change data capture to synchronize real-time database changes into warehouses and lakes.
- [External Storage Integrations](https://awesome-repositories.com/f/data-databases/external-storage-integrations.md) — Enables linking and importing diverse external database engines and storage systems for query development. ([source](http://www.dinky.org.cn/docs/1.0/get_started/overview))
- [Multi-Tenant Resource Isolation](https://awesome-repositories.com/f/data-databases/multi-tenant-resource-isolation.md) — Organizes users and workloads into namespaces with role-based access control to secure shared cluster environments.
- [Real-Time Data Integration Platforms](https://awesome-repositories.com/f/data-databases/real-time-data-integration-platforms.md) — Provides a platform for synchronizing live data between heterogeneous databases and analytical warehouses with automatic schema evolution.
- [Real-time Data Synchronization](https://awesome-repositories.com/f/data-databases/real-time-data-synchronization.md) — Captures database changes in real-time to automate table creation and schema evolution in target warehouses. ([source](http://www.dinky.org.cn/docs/next/get_started/overview))
- [Streaming Pipeline Orchestrators](https://awesome-repositories.com/f/data-databases/real-time-data-synchronization/streaming-pipeline-orchestrators.md) — Orchestrates the complete lifecycle of streaming jobs, covering deployment, external synchronization, and state recovery.
- [Change Filters](https://awesome-repositories.com/f/data-databases/change-data-capture/change-filters.md) — Excludes specific database operations from the synchronization stream using integrated configuration parameters. ([source](http://www.dinky.org.cn/docs/next/practical_guide/cdcsource_practice/cdcsource_overview))
- [Full Instance Synchronization](https://awesome-repositories.com/f/data-databases/change-data-capture/full-instance-synchronization.md) — Builds real-time data pipelines to synchronize all tables from a complete database instance to downstream systems. ([source](http://www.dinky.org.cn/docs/next/practical_guide/cdcsource_practice/cdcsource_overview))
- [Data Lineage Visualizations](https://awesome-repositories.com/f/data-databases/data-lineage-visualizations.md) — Maps dependencies between data elements to analyze the global impact of changes across the pipeline. ([source](http://www.dinky.org.cn/docs/next/roadmap))
- [Stream Snapshot Restorers](https://awesome-repositories.com/f/data-databases/data-restoration-tools/stream-snapshot-restorers.md) — Includes utilities to restore streaming application configurations and message history from stream-specific snapshots. ([source](http://www.dinky.org.cn/docs/next/get_started/overview))
- [Multi-Destination Data Routing](https://awesome-repositories.com/f/data-databases/multi-destination-data-routing.md) — Provides mechanisms for directing processed data streams to multiple external output targets and clusters. ([source](http://www.dinky.org.cn/docs/next/practical_guide/cdcsource_practice/cdcsource_overview))
- [Pluggable Connector Frameworks](https://awesome-repositories.com/f/data-databases/pluggable-storage-drivers/pluggable-connector-frameworks.md) — Integrates diverse data sources and destinations through a standardized, pluggable connector architecture.
- [Source-to-Sink Table Mappings](https://awesome-repositories.com/f/data-databases/source-to-sink-table-mappings.md) — Implements rules for matching and renaming source tables to destination tables using prefixes and suffixes. ([source](http://www.dinky.org.cn/docs/next/practical_guide/cdcsource_practice/cdcsource_overview))
- [Automated Sink Schema Generation](https://awesome-repositories.com/f/data-databases/source-to-sink-table-mappings/automated-sink-schema-generation.md) — Automatically constructs destination table definitions by extracting metadata from source stages during synchronization.
- [Custom SQL Functions](https://awesome-repositories.com/f/data-databases/sql-aggregate-functions/custom-sql-functions.md) — Allows development and submission of custom SQL functions to handle specialized data processing tasks. ([source](http://www.dinky.org.cn/docs/1.0/get_started/overview))
- [User-Defined Function Development](https://awesome-repositories.com/f/data-databases/sql-aggregate-functions/custom-sql-functions/user-defined-function-development.md) — Enables writing and submitting custom processing logic using common programming languages to extend data processing. ([source](http://www.dinky.org.cn/docs/next/get_started/overview))
- [SQL Schema Generators](https://awesome-repositories.com/f/data-databases/sql-schema-generators.md) — Automatically derives destination SQL table and type definitions by capturing metadata from the source stage. ([source](http://www.dinky.org.cn/docs/next/practical_guide/cdcsource_practice/cdcsource_overview))
- [Streaming State Recovery](https://awesome-repositories.com/f/data-databases/undo-logs/transaction-recovery-logs/log-state-recovery/streaming-state-recovery.md) — Recovers incremental operator states and results from durable storage to ensure streaming stability after failures. ([source](http://www.dinky.org.cn/docs/1.0/get_started/overview))

### Development Tools & Productivity

- [SQL Development Environments](https://awesome-repositories.com/f/development-tools-productivity/sql-development-environments.md) — Provides a web-based integrated environment for writing, verifying, and optimizing SQL streaming applications. ([source](https://cdn.jsdelivr.net/gh/datalinkdc/dinky@dev/README.md))
- [Streaming State Recovery](https://awesome-repositories.com/f/development-tools-productivity/crash-recovery-systems/checkpoint-based-recovery/streaming-state-recovery.md) — Restores streaming applications to specific historical points by managing checkpoints and savepoints within the cluster.
- [Streaming Application Authoring](https://awesome-repositories.com/f/development-tools-productivity/streaming-application-authoring.md) — Ships a lightweight code editor with completion and logic checking specifically for authoring data streaming jobs. ([source](http://www.dinky.org.cn/))
- [Stream Processing Job Submissions](https://awesome-repositories.com/f/development-tools-productivity/job-scheduling-resources/containerized-batch-job-schedulers/stream-processing-job-submissions.md) — Submits stream processing pipelines to clusters using SQL statements and a managed interface. ([source](http://www.dinky.org.cn/docs/next/get_started/overview))

### Artificial Intelligence & ML

- [Unified Metadata Catalogs](https://awesome-repositories.com/f/artificial-intelligence-ml/unified-data-querying/unified-metadata-catalogs.md) — Provides a centralized interface to manage schemas and connection details across diverse external database engines.
- [SQL Lineage Analyzers](https://awesome-repositories.com/f/artificial-intelligence-ml/data-lineage/automated-lineage-capturers/sql-lineage-analyzers.md) — Analyzes SQL statements to track column-level data lineage and visualize transformation logic. ([source](http://www.dinky.org.cn/))
- [Lineage Mapping](https://awesome-repositories.com/f/artificial-intelligence-ml/data-lineage/lineage-mapping.md) — Tracks the flow of data through applications to visualize transformations and analyze the impact of changes.

### Part of an Awesome List

- [Streaming SQL IDEs](https://awesome-repositories.com/f/awesome-lists/devtools/sql-clients-and-ides/streaming-sql-ides.md) — Provides a web-based IDE for authoring streaming SQL with integrated logic plan visualization and lineage tracking.

### DevOps & Infrastructure

- [Centralized Environment Management](https://awesome-repositories.com/f/devops-infrastructure/cluster-environment-automation/centralized-environment-management.md) — Registers and controls multiple remote processing clusters from a single centralized interface.
- [Distributed Cluster Management](https://awesome-repositories.com/f/devops-infrastructure/distributed-cluster-management.md) — Provides a centralized interface for the administration and orchestration of shared computing resources across multiple processing clusters.
- [Cluster Administration](https://awesome-repositories.com/f/devops-infrastructure/infrastructure/cluster-service-orchestration/cluster-administration.md) — Organizes configurations, data sources, and global variables within a centralized cluster management interface. ([source](https://cdn.jsdelivr.net/gh/datalinkdc/dinky@dev/README.md))
- [Streaming Cluster Orchestration](https://awesome-repositories.com/f/devops-infrastructure/kubernetes-cluster-management/multi-cluster-deployment-managers/database-cluster-deployments/streaming-cluster-orchestration.md) — Orchestrates the full lifecycle of distributed streaming clusters, including deployment and resource administration.
- [Cluster Job Launchers](https://awesome-repositories.com/f/devops-infrastructure/cluster-job-launchers.md) — Dispatches developed streaming applications to remote compute resources on production clusters for persistent execution. ([source](http://www.dinky.org.cn/docs/next/get_started/quick_experience))
- [External Cluster Imports](https://awesome-repositories.com/f/devops-infrastructure/cluster-management-connectivity/external-cluster-imports.md) — Connects external processing clusters to a management platform by specifying names and addresses. ([source](http://www.dinky.org.cn/docs/next/user_guide/studio/cli))
- [SQL Project Deployments](https://awesome-repositories.com/f/devops-infrastructure/data-project-deployments/sql-project-deployments.md) — Deploys declarative SQL projects and tasks across diverse environments and session modes. ([source](https://cdn.jsdelivr.net/gh/datalinkdc/dinky@dev/README.md))
- [Streaming Job Execution](https://awesome-repositories.com/f/devops-infrastructure/job-execution-engines/kubernetes-job-execution/streaming-job-execution.md) — Deploys streaming tasks across multiple session or application modes in diverse environments. ([source](http://www.dinky.org.cn/docs/1.0/get_started/overview))
- [Stream Processing Pipeline Deployments](https://awesome-repositories.com/f/devops-infrastructure/stream-processing-pipeline-deployments.md) — Deploys and monitors fault-tolerant stream processing applications across local and distributed environments. ([source](http://www.dinky.org.cn/docs/1.1/get_started/overview))

### Software Engineering & Architecture

- [Job Lifecycle Management](https://awesome-repositories.com/f/software-engineering-architecture/background-job-dispatchers/job-lifecycle-management.md) — Provides full lifecycle controls for streaming tasks, including triggering starts, canceling processes, and managing recovery points. ([source](http://www.dinky.org.cn/docs/next/openapi/openapi_overview))
- [Stateful Recovery Mechanisms](https://awesome-repositories.com/f/software-engineering-architecture/fault-tolerance-strategies/fault-tolerance-implementation/stateful-recovery-mechanisms.md) — Implements stateful recovery mechanisms using snapshots and checkpoints to restore streaming jobs without data loss. ([source](http://www.dinky.org.cn/docs/1.1/get_started/overview))
- [Execution Configurations](https://awesome-repositories.com/f/software-engineering-architecture/node-runtime-configuration/workflow-node-form-management/ai-workflow-node-configurations/sql-editors/execution-configurations.md) — Defines execution modes and parallelism within a managed editor to configure streaming applications. ([source](http://www.dinky.org.cn/docs/next/get_started/quick_experience))

### System Administration & Monitoring

- [Data Pipeline Lineage Inspectors](https://awesome-repositories.com/f/system-administration-monitoring/pipeline-observability-tools/data-pipeline-lineage-inspectors.md) — Tracks data lineage and visualizes transformation logic to provide deep observability into data pipelines.
- [Real-Time Application Performance Monitors](https://awesome-repositories.com/f/system-administration-monitoring/application-performance-monitoring/application-performance-monitoring/real-time-application-performance-monitors.md) — Provides live analysis of streaming application performance and health via real-time monitoring views. ([source](http://www.dinky.org.cn/docs/1.0/get_started/overview))
- [Cluster Health Monitoring](https://awesome-repositories.com/f/system-administration-monitoring/cluster-health-monitoring.md) — Provides interfaces for querying the status and health metrics of distributed processing clusters. ([source](http://www.dinky.org.cn/docs/1.1/get_started/overview))
- [Catalog Management](https://awesome-repositories.com/f/system-administration-monitoring/cluster-management/catalog-management.md) — Provides a unified interface to configure and manage multiple independent database catalogs for simultaneous querying. ([source](http://www.dinky.org.cn/docs/next/get_started/overview))
- [Cluster Stability Monitoring](https://awesome-repositories.com/f/system-administration-monitoring/cluster-stability-monitoring.md) — Monitors runtime information and cluster logs while managing snapshot recovery to maintain system stability. ([source](http://www.dinky.org.cn/))
- [Job Execution Tracking](https://awesome-repositories.com/f/system-administration-monitoring/job-execution-tracking.md) — Retrieves operational data and execution plans to track the lifecycle and processing logic of computational tasks. ([source](http://www.dinky.org.cn/docs/next/openapi/openapi_overview))
- [Real-Time Application Log Monitoring](https://awesome-repositories.com/f/system-administration-monitoring/real-time-application-log-monitoring.md) — Implements real-time log monitoring and online controls to manage the operational lifecycle of streaming jobs. ([source](https://cdn.jsdelivr.net/gh/datalinkdc/dinky@dev/README.md))
- [Real-Time Monitoring](https://awesome-repositories.com/f/system-administration-monitoring/real-time-monitoring.md) — Observes live cluster health and exception logs in real-time to maintain streaming stability. ([source](http://www.dinky.org.cn/docs/next/get_started/overview))

### Security & Cryptography

- [Enterprise Data Governance](https://awesome-repositories.com/f/security-cryptography/enterprise-data-governance.md) — Controls system access through multi-tenancy, role-based permissions, and centralized identity authentication.
- [Enterprise Security Integrations](https://awesome-repositories.com/f/security-cryptography/identity-access-management/authentication-strategies/authorization-and-user-administration/enterprise-security-integrations.md) — Integrates identity verification protocols to secure access to resource centers for enterprise-grade security. ([source](http://www.dinky.org.cn/docs/next/roadmap))
- [Multi-Tenant Administration](https://awesome-repositories.com/f/security-cryptography/multi-tenant-administration.md) — Provides administrative management of isolated namespaces, roles, and multi-tenancy configurations. ([source](http://www.dinky.org.cn/docs/1.0/get_started/overview))
- [Role-Based Access Controls](https://awesome-repositories.com/f/security-cryptography/multi-tenant-isolation/role-based-access-controls.md) — Implements role-based access controls and multi-tenant isolation to manage system usage and permissions. ([source](https://cdn.jsdelivr.net/gh/datalinkdc/dinky@dev/README.md))
- [Single Sign-On Integrations](https://awesome-repositories.com/f/security-cryptography/single-sign-on-integrations.md) — Centralizes user authentication through single sign-on integrations for access across multiple services. ([source](http://www.dinky.org.cn/docs/next/roadmap))

### User Interface & Experience

- [Query Result Previews](https://awesome-repositories.com/f/user-interface-experience/window-previews/query-result-previews.md) — Runs select statements in a local environment to preview data output and verify logic before deployment. ([source](http://www.dinky.org.cn/docs/next/get_started/quick_experience))