Awesome GitHub RepositoriesSchema Registries

Zod is a TypeScript-first schema declaration and validation library designed to ensure end-to-end data integrity. It functions as a runtime type guard, allowing developers to define complex data structures through a declarative, chainable syntax. By using these schema definitions, the library automatically derives static TypeScript types, eliminating the need for manual type duplication and ensuring that runtime data matches expected application contracts. The library distinguishes itself through functional schema composition, which enables the creation of hierarchical structures by nesting a

RocketMQ is a cloud-native distributed messaging platform and streaming engine. It functions as a distributed transactional queue that ensures atomicity between local transactions and message delivery, and serves as an MQTT IoT message broker to bridge lightweight device traffic into high-performance data streams. The system is distinguished by a Kubernetes-native architecture that decouples compute from storage to allow independent scaling of traffic and data retention. It utilizes a tiered storage model to offload older data to remote storage and employs quorum-based replication and automat

Connect is a Kafka data integration platform and stream processing engine used to build declarative pipelines that move and transform messages between Kafka topics and external sources. It functions as a Kafka Connect framework and a change data capture tool, streaming real-time database modifications to synchronize data across distributed environments. The project differentiates itself through a dedicated mapping language for mutating and reshaping message payloads and the ability to execute custom processing logic within a sandboxed WebAssembly runtime. It also provides an observability pip

Materialize is a streaming SQL database that continuously ingests live data from sources such as Kafka, Redpanda, PostgreSQL, and MySQL, and incrementally maintains materialized views. It provides a PostgreSQL-compatible query engine that accepts standard SQL over the PostgreSQL wire protocol, enabling any existing SQL client or BI tool to query real-time data. The system also includes a Model Context Protocol (MCP) server that exposes live materialized view data to AI agents, providing fresh context without polling. Materialize distinguishes itself through its ability to offer configurable c

Kafdrop ist eine webbasierte Schnittstelle zur Überwachung und Verwaltung von Apache Kafka-Clustern, Topics, Brokern und Consumer-Groups. Es fungiert als Cluster-Monitor und Topic-Manager, der eine visuelle Darstellung des Broker-Status, der Partitionszuweisungen und des Consumer-Group-Lags bietet. Das System enthält einen Message-Browser, der Nachrichten aus Kafka-Topics lesen, dekodieren und veröffentlichen kann, wobei Schema-Registries oder Deskriptordateien verwendet werden. Es bietet zudem eine Metadaten-API, die Cluster-Informationen über JSON-Endpunkte für die Integration in externe Monitoring-Tools bereitstellt. Die Plattform deckt administrative Bereiche wie das Topic-Lifecycle-Management, Sicherheitsaudits von Access Control Lists (ACLs) und den Aufbau verschlüsselter Broker-Verbindungen mittels TLS-Zertifikaten und SASL-Anmeldedaten ab. Die Observability-Funktionen umfassen die Verfolgung von Offsets pro Partition und die Berechnung des Consumer-Group-Lags.

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

kcat ist ein Kommandozeilen-Client für Apache Kafka, der verwendet wird, um Nachrichten unter Verwendung des nativen Wire-Protokolls zu produzieren, zu konsumieren und zu debuggen. Er bietet eine Reihe von Tools für die Interaktion mit Kafka-Clustern, einschließlich eines Protokoll-Debuggers zur Untersuchung von Cluster-Metadaten und eines Transaktionsmanagers für die Handhabung atomarer Nachrichten-Batches. Das Projekt verfügt über einen spezialisierten Avro-Schema-Dekoder, der binär kodierte Nachrichten in menschenlesbares JSON umwandelt, indem er sich in Remote-Schema-Registries oder lokale Dateien integriert. Zusätzlich enthält es einen In-Memory-Simulator, der das Testen von Producer- und Consumer-Logik durch die Simulation ephemerer Broker-Verhaltensweisen ermöglicht, ohne dass eine externe Infrastruktur erforderlich ist. Das Toolset deckt ein breites Spektrum an Messaging-Operationen ab, einschließlich Unterstützung für ausbalancierte Consumer-Gruppen, zeitstempelbasierte Offset-Suche und transaktionales Daten-Streaming von der Standardeingabe. Zudem bietet es Dienstprogramme für die Konfiguration der Verbindungssicherheit und die Untersuchung von Cluster-Metadaten.

AKHQ is a web-based management interface for Apache Kafka, providing a centralized platform for administering clusters, topics, and consumer groups. It serves as a comprehensive monitoring and administration tool that includes a Kafka Connect manager and a ksqlDB administration interface. The platform distinguishes itself through extensive schema registry integration, allowing users to browse and decode Avro, Protobuf, and JSON messages using Confluent, Tibco, or AWS Glue registries. It also features a granular security model with role-based access control, sensitive data masking, and support

franz-go is a low-level Go client library and wire protocol implementation for producing, consuming, and administering Kafka clusters. It functions as a zero-allocation network driver that utilizes a direct TCP communication layer to handle requests and responses. The project integrates a schema registry client for encoding and decoding structured data. It provides a programmatic interface for cluster administration, including the management of topics, access control lists, and broker configurations. The library covers data consumption through consumer groups, message production with transac