22 repositorios
Mathematical strategies for distributing data across partitions, such as hash-based or range-based.
Distinguishing note: Focuses on the algorithms used for partitioning, distinct from the general concept of partitioning.
Explore 22 awesome GitHub repositories matching data & databases · Partitioning Algorithms. Refine with filters or upvote what's useful.
This project is a comprehensive educational resource focused on the principles, patterns, and trade-offs required to design scalable, reliable, and high-performance distributed systems. It provides a structured curriculum that covers the fundamental architectural strategies necessary for building modern software infrastructure, ranging from high-level system decomposition to low-level networking and data management. The repository distinguishes itself by offering deep dives into complex architectural patterns, such as microservices-based decomposition, event-driven communication, and command-
Details strategies like hash-based partitioning for distributing data.
This project is a comprehensive collection of common computer science algorithms and data structures implemented in Swift. It serves as an educational reference and library for studying computational complexity, algorithmic logic, and data structure engineering through practical code examples. The repository provides a wide suite of data structure implementations, including various types of linked lists, heaps, hash tables, and an extensive range of hierarchical trees such as Red-Black, B-Tree, and Splay trees. It also covers diverse sorting and searching techniques, from basic bubble sort to
Implements efficient array partitioning around pivots to facilitate sorting operations.
This project is a comprehensive, community-maintained knowledge base and toolkit designed for competitive programming. It serves as a centralized repository for algorithmic theory, data structures, and mathematical techniques, providing a structured reference for informatics and collegiate programming competitions. The project distinguishes itself by integrating educational content with a robust suite of automation utilities. It provides a complete workflow for competitive programming, including tools for automated test case generation, solution verification, and direct interaction with onlin
Organizes elements by recursively partitioning a sequence around a pivot value to ensure sub-sequences are ordered.
TimescaleDB is an open-source PostgreSQL extension that adds native time-series capabilities to the database. At its core, it transforms standard PostgreSQL tables into hypertables—automatically partitioned by time intervals—so data is stored in fixed-size chunks without manual sharding. The extension includes a library of over 200 built-in SQL functions purpose-built for time-series workloads, such as time bucketing, gap filling, percentile estimation, and time-weighted averages. What distinguishes TimescaleDB from generic PostgreSQL is its set of integrated time-series features that work th
Splits large tables into smaller partitions based on time or a key to improve query performance.
This library is a collection of generic utilities for the Go programming language designed to simplify the manipulation of slices and maps. It provides a functional toolkit that enables developers to perform data transformations, such as filtering, mapping, and reducing, while maintaining strict type safety through the use of language-level generics. The project distinguishes itself by offering a dual approach to data processing that balances functional programming patterns with performance-oriented execution. It supports both immutable functional pipelines for predictable state transitions a
Provides utilities for grouping, chunking, and partitioning complex data collections.
Bullet3 is a professional physics simulation engine designed for calculating rigid body, soft body, and collision dynamics within 3D environments and robotics applications. It functions as a computational framework for determining complex geometric intersections and contact manifolds between objects in simulated space. The library distinguishes itself through a distributed rendering framework that scales heavy graphical workloads and scene generation tasks across large clusters of machines. This capability enables the production of massive datasets by distributing complex scene generation acr
Groups independent clusters of interacting objects into isolated islands to allow parallel processing of physics updates.
Dask es un framework de computación paralela y un programador de tareas distribuido diseñado para escalar flujos de trabajo de ciencia de datos en Python desde máquinas individuales hasta grandes clústeres. Funciona como un gestor de recursos de clúster que orquesta la lógica computacional representando las tareas y sus dependencias como grafos acíclicos dirigidos. Esta arquitectura permite al sistema automatizar la distribución de cargas de trabajo a través del hardware disponible mientras gestiona requisitos de ejecución complejos. El proyecto se distingue por un motor de evaluación perezosa que difiere las operaciones de datos hasta que se solicitan explícitamente, permitiendo la optimización global del grafo y una asignación eficiente de recursos. Incorpora el volcado de datos consciente de la memoria para evitar fallos del sistema al procesar conjuntos de datos que exceden la memoria disponible, y utiliza la fusión de grafos de tareas para combinar secuencias de operaciones en pasos de ejecución únicos, minimizando la sobrecarga de programación y la comunicación entre nodos. La plataforma proporciona una superficie de capacidades integral para el análisis de datos a gran escala, incluyendo soporte para aprendizaje automático distribuido, integración de computación de alto rendimiento y procesamiento de datos en paralelo. Ofrece herramientas extensas para la gestión del ciclo de vida del clúster, perfilado de rendimiento y monitoreo en tiempo real de la ejecución de tareas. Los usuarios pueden desplegar estos entornos en diversas infraestructuras, incluyendo hardware local, proveedores de nube, sistemas en contenedores y clústeres de computación de alto rendimiento.
Adjusts the number of output partitions during grouping operations to balance memory usage and computational efficiency.
Citus is a PostgreSQL extension that transforms a standard database into a distributed system. It functions as a sharding framework and distributed SQL engine, enabling horizontal scaling by partitioning tables across a cluster of nodes. By utilizing a coordinator-worker topology, the system manages metadata and routes queries to the appropriate nodes, allowing for parallel execution of complex operations across distributed data shards. The platform distinguishes itself through its specialized support for multi-tenant architectures and real-time analytical processing. It enables tenant-based
Automates the creation and removal of time-based table partitions using scheduled maintenance tasks to ensure continuous data ingestion and efficient retention.
Mac app that shows all open files, directories, sockets, pipes and devices in use by all running processes. Nice GUI for lsof.
Filters the displayed list by name, access mode, volume, type, location, or regular expression to narrow results.
pgloader is a command-line tool that automates the migration of data and schema from various source databases and file formats into PostgreSQL. It combines schema discovery, parallel data pipelines, and type casting into a single, declarative workflow, using PostgreSQL's COPY protocol for high-throughput bulk loading. The tool distinguishes itself by compiling a dedicated command language into concurrent reader-writer pipelines that handle schema introspection, data transformation, and error-resilient batch processing. It supports migrating entire databases from MySQL, MS SQL, SQLite, and Pos
Selectively includes or excludes tables from migration using name lists or regular expressions.
Implements spatial hashing and distance-based interest management to reduce network traffic in multiplayer games.
GreptimeDB is a distributed, open-source time-series database built for unified observability. It stores and queries metrics, logs, and traces together in a single columnar engine, supporting both SQL and PromQL for analysis. The database is designed as a Kubernetes-native operator with a decoupled compute and storage architecture, enabling horizontal scaling and multi-region deployment. What distinguishes GreptimeDB is its role as a multi-protocol ingestion gateway, accepting data through OpenTelemetry, Prometheus Remote Write, InfluxDB, Loki, Elasticsearch, Kafka, and MQTT protocols without
Lists tables in a database with optional name filtering for discovery.
CodeIgniter is a PHP web framework built on the Model-View-Controller pattern, designed for building full-stack web applications. It provides a lightweight toolkit with minimal configuration, organizing application logic into controllers, models, and views for clean separation of concerns. The framework includes a fluent query builder for constructing SQL statements programmatically, PSR-4 autoloading with namespace mapping, and a service-based dependency injection container for managing shared class instances. The framework distinguishes itself through its comprehensive set of built-in tools
Provides a utility method to enumerate all table names in the currently connected database.
Cortex is an open-source, horizontally scalable metrics platform that ingests, stores, and queries Prometheus-compatible time-series data with multi-tenant isolation. It accepts metrics via Prometheus remote write and OpenTelemetry, executes PromQL queries against both recent and historical data, and provides a Prometheus-compatible alerting and recording rule engine with an integrated Alertmanager. The system is built as a set of independently scalable microservices that use hash-ring-based sharding, gossip-based cluster membership, and tenant-aware object storage to distribute workloads acro
Divides timeseries across multiple sub-blocks using a hash of their labels for parallel compaction.
FastStream is an asynchronous Python framework designed for building event-driven microservices. It provides a unified abstraction layer for interacting with various message brokers, enabling developers to manage event production and consumption through a consistent interface while maintaining access to native provider-specific features. The framework centers on a decorator-based routing model that binds application logic directly to broker topics, supported by a built-in dependency injection container that resolves resources at runtime. The framework distinguishes itself through its deep int
Supports routing messages by partition keys to ensure data grouping and processing order.
This repository is a comprehensive collection of fully worked solutions to exercises and problems from the standard algorithms textbook by Cormen, Leiserson, Rivest, and Stein (CLRS). It serves as an educational reference for algorithm design and analysis, providing step-by-step reasoning, pseudocode, and mathematical proofs for a wide range of topics. The content spans core computer science areas: algorithm analysis with asymptotic notation, recurrence solving, and amortized cost analysis; data structure implementation and operations for binary search trees, red-black trees, B-trees, Fibonac
Illustrates the quicksort partition subroutine including pivot selection and array rearrangement.
Jitsu es una plataforma de datos de clientes diseñada para recopilar, transformar y enrutar eventos de aplicaciones a almacenes de datos y herramientas de marketing. Funciona como un motor de ingesta de eventos y enrutador de almacenes de datos, capturando datos de comportamiento a través de API y SDK para su procesamiento y almacenamiento en tiempo real. La plataforma cuenta con un pipeline de datos programable en JavaScript que permite el filtrado, enriquecimiento y remodelación de los datos de eventos durante el tránsito. Incluye un sistema de unión de identidades de clientes que fusiona identificadores de usuarios anónimos y conocidos para mantener perfiles de clientes persistentes dentro de un almacén. El sistema cubre una amplia gama de capacidades, incluyendo la recopilación de eventos de múltiples fuentes desde entornos web y móviles, la evolución automática de esquemas para almacenes de destino y el enrutamiento a múltiples destinos hacia plataformas SaaS y bases de datos SQL. Proporciona un conjunto de herramientas para desarrolladores para probar la lógica de transformación y admite el despliegue a través de Kubernetes o entornos Docker autohospedados.
Organizes records into time-based segments using a timestamp column to speed up analytical queries and reduce data scanned during lookups.
GAM is a command-line tool for administering Google Workspace and Cloud Identity. It translates command-line arguments into structured API calls, enabling administrators to manage users, groups, organizational units, and domain settings across a Google Workspace environment. The tool handles authentication through OAuth2 flows, service accounts, and workload identity federation, and supports multi-tenant configurations for managing multiple domains or cloud projects from a single installation. GAM distinguishes itself through its batch processing and automation capabilities. It can process la
Filters users, groups, files, or messages using regular expressions based on name or email.
This repository is a curated guide and implementation library of coding patterns used to solve data structures and algorithms problems. It serves as a technical interview study resource, providing a comprehensive set of strategies and computational logic examples for optimizing time and space complexity. The project focuses on standardized algorithmic patterns, including sliding windows, two pointers, and dynamic programming. It features specific implementations for a wide range of challenges, such as LeetCode problem solutions and specialized techniques like cyclic sort and bitwise XOR opera
Implements array partitioning that organizes three distinct values into separate zones in a single pass.
Screw es una herramienta de análisis de esquemas de bases de datos utilizada para generar automáticamente documentación técnica y objetos de datos en Java. Escanea tablas de bases de datos para exportar sus definiciones, transformando los metadatos del esquema en informes estructurados y código de aplicación. El proyecto genera objetos Java simples (POJOs) basados en las estructuras de tablas existentes para automatizar el modelado de datos. También produce documentación del esquema de la base de datos en formatos HTML, Word y Markdown. El sistema incluye capacidades para el filtrado de exportación de tablas mediante patrones de nombres, extracción de esquemas basada en JDBC y generación de código basada en plantillas para mapear tipos de columnas de bases de datos a tipos del lenguaje Java.
Provides the ability to include or exclude specific database tables from the export process using name patterns.