7 个仓库
Utilities for transforming, aggregating, and analyzing raw data streams.
Distinguishing note: Focuses on server-side computation, distinct from client-side event collection.
Explore 7 awesome GitHub repositories matching data & databases · Data Processing. Refine with filters or upvote what's useful.
Umami is a self-hosted, privacy-focused web analytics platform designed to provide full control over infrastructure and user data. It captures website traffic and visitor behavior through anonymous tracking methods that avoid cookies, browser fingerprinting, and the storage of personally identifiable information. The platform distinguishes itself through a comprehensive suite of behavioral analysis tools, including session replays, heatmaps, and cohort-based retention reporting. It features a multi-tenant architecture that allows teams to manage multiple websites within a single, collaborativ
Aggregates raw event logs into meaningful insights on the server to minimize client-side overhead.
Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network
Performs local data aggregation to reduce network traffic and compute load before forwarding to global nodes.
This project provides a comprehensive guide to architectural patterns and best practices for building scalable, maintainable, and performant web applications using FastAPI. It focuses on standardizing development approaches for Python web services, emphasizing robust request validation, dependency injection, and automated documentation standards to ensure consistent API design. The guide distinguishes itself by promoting domain-driven modular packaging, which organizes application logic into isolated, feature-based directories to support long-term codebase scalability. It also details strateg
Performs complex data joins and aggregations directly within the database engine for native performance.
This project is a high-performance MQTT broker and IoT data platform designed to manage millions of concurrent device connections. It provides a scalable infrastructure for ingesting, processing, and routing telemetry data across distributed systems, utilizing an actor-based concurrency model to maintain high availability and state synchronization across cluster nodes. The platform distinguishes itself through integrated stream processing and edge computing capabilities. It allows users to execute declarative SQL-based rules directly against incoming message streams for real-time filtering, t
Filters, aggregates, and transforms data streams locally to reduce bandwidth consumption and enable low-latency responses.
Boto3 is the AWS SDK for Python, providing a programmatic interface for managing and automating AWS cloud infrastructure and services. It serves as a cloud management API client and resource manager for provisioning, configuring, and scaling virtual servers, databases, and storage. The library enables the implementation of infrastructure-as-code through declarative templates and scripts, allowing for the deployment of identical resource stacks across multiple accounts and geographic regions. It also provides a framework for coordinating distributed workflows, serverless functions, and contain
Runs custom serverless code during object requests to filter or modify data in real-time.
lakeFS 是一个数据湖版本控制系统,为存储在对象存储中的大型数据集提供类似 Git 的分支和提交功能。它作为一个版本控制层,支持创建不可变快照、原子提交和零拷贝分支,从而在不复制物理文件的情况下为数据实验创建隔离环境。 该系统充当 S3 兼容的存储网关和 Iceberg REST 目录,允许标准云存储协议和兼容客户端管理版本化表。它通过使用事件驱动的钩子系统在更改合并到生产环境之前根据治理策略验证数据集,从而充当数据质量守门人。 该平台涵盖了广泛的数据治理功能,包括 Pull Request 协作、基于角色的访问控制和数据血缘追踪。它为工作流编排、机器学习管线和各种大数据计算引擎提供了集成,支持多云存储连接以及通过 SSO 和 SCIM 进行身份同步。 该软件可以使用二进制文件、容器或 Helm Chart 安装,以便在 Kubernetes 上部署。
Updates embeddings by processing only the added, removed, or modified data between two commits.
This project is a C++ learning resource and study guide consisting of structured notes and programming examples. It provides practical implementations and exercise solutions covering core language syntax, data types, and control flow. The repository features specialized samples for object-oriented design, including class inheritance, polymorphism, and abstract classes. It includes demonstrations of memory management techniques such as dynamic allocation, move semantics, and placement new, as well as template programming examples for creating generic functions and data structures. The codebas
Implements logic to aggregate and calculate totals from multidimensional grid-based data structures.