22 dépôts
Mathematical strategies for distributing data across partitions, such as hash-based or range-based.
Distinguishing note: Focuses on the algorithms used for partitioning, distinct from the general concept of partitioning.
Explore 22 awesome GitHub repositories matching data & databases · Partitioning Algorithms. Refine with filters or upvote what's useful.
This project is a comprehensive educational resource focused on the principles, patterns, and trade-offs required to design scalable, reliable, and high-performance distributed systems. It provides a structured curriculum that covers the fundamental architectural strategies necessary for building modern software infrastructure, ranging from high-level system decomposition to low-level networking and data management. The repository distinguishes itself by offering deep dives into complex architectural patterns, such as microservices-based decomposition, event-driven communication, and command-
Details strategies like hash-based partitioning for distributing data.
This project is a comprehensive collection of common computer science algorithms and data structures implemented in Swift. It serves as an educational reference and library for studying computational complexity, algorithmic logic, and data structure engineering through practical code examples. The repository provides a wide suite of data structure implementations, including various types of linked lists, heaps, hash tables, and an extensive range of hierarchical trees such as Red-Black, B-Tree, and Splay trees. It also covers diverse sorting and searching techniques, from basic bubble sort to
Implements efficient array partitioning around pivots to facilitate sorting operations.
This project is a comprehensive, community-maintained knowledge base and toolkit designed for competitive programming. It serves as a centralized repository for algorithmic theory, data structures, and mathematical techniques, providing a structured reference for informatics and collegiate programming competitions. The project distinguishes itself by integrating educational content with a robust suite of automation utilities. It provides a complete workflow for competitive programming, including tools for automated test case generation, solution verification, and direct interaction with onlin
Organizes elements by recursively partitioning a sequence around a pivot value to ensure sub-sequences are ordered.
TimescaleDB is an open-source PostgreSQL extension that adds native time-series capabilities to the database. At its core, it transforms standard PostgreSQL tables into hypertables—automatically partitioned by time intervals—so data is stored in fixed-size chunks without manual sharding. The extension includes a library of over 200 built-in SQL functions purpose-built for time-series workloads, such as time bucketing, gap filling, percentile estimation, and time-weighted averages. What distinguishes TimescaleDB from generic PostgreSQL is its set of integrated time-series features that work th
Splits large tables into smaller partitions based on time or a key to improve query performance.
This library is a collection of generic utilities for the Go programming language designed to simplify the manipulation of slices and maps. It provides a functional toolkit that enables developers to perform data transformations, such as filtering, mapping, and reducing, while maintaining strict type safety through the use of language-level generics. The project distinguishes itself by offering a dual approach to data processing that balances functional programming patterns with performance-oriented execution. It supports both immutable functional pipelines for predictable state transitions a
Provides utilities for grouping, chunking, and partitioning complex data collections.
Bullet3 is a professional physics simulation engine designed for calculating rigid body, soft body, and collision dynamics within 3D environments and robotics applications. It functions as a computational framework for determining complex geometric intersections and contact manifolds between objects in simulated space. The library distinguishes itself through a distributed rendering framework that scales heavy graphical workloads and scene generation tasks across large clusters of machines. This capability enables the production of massive datasets by distributing complex scene generation acr
Groups independent clusters of interacting objects into isolated islands to allow parallel processing of physics updates.
Dask est un framework de calcul parallèle et un planificateur de tâches distribué conçu pour mettre à l'échelle les flux de travail de science des données Python, des machines uniques aux grands clusters. Il fonctionne comme un gestionnaire de ressources de cluster qui orchestre la logique computationnelle en représentant les tâches et leurs dépendances sous forme de graphes acycliques dirigés. Cette architecture permet au système d'automatiser la distribution des charges de travail sur le matériel disponible tout en gérant des exigences d'exécution complexes. Le projet se distingue par un moteur d'évaluation paresseuse qui diffère les opérations sur les données jusqu'à ce qu'elles soient explicitement demandées, permettant une optimisation globale du graphe et une allocation efficace des ressources. Il intègre le déversement de données conscient de la mémoire pour éviter les plantages du système lors du traitement de jeux de données dépassant la mémoire disponible, et il utilise la fusion de graphes de tâches pour combiner des séquences d'opérations en étapes d'exécution uniques, minimisant la surcharge de planification et la communication entre nœuds. La plateforme fournit une surface de capacités complète pour l'analyse de données à grande échelle, incluant le support pour l'apprentissage automatique distribué, l'intégration du calcul haute performance et le traitement de données parallèle. Elle offre des outils étendus pour la gestion du cycle de vie des clusters, le profilage des performances et la surveillance en temps réel de l'exécution des tâches. Les utilisateurs peuvent déployer ces environnements sur diverses infrastructures, incluant le matériel local, les fournisseurs cloud, les systèmes conteneurisés et les clusters de calcul haute performance.
Adjusts the number of output partitions during grouping operations to balance memory usage and computational efficiency.
Citus is a PostgreSQL extension that transforms a standard database into a distributed system. It functions as a sharding framework and distributed SQL engine, enabling horizontal scaling by partitioning tables across a cluster of nodes. By utilizing a coordinator-worker topology, the system manages metadata and routes queries to the appropriate nodes, allowing for parallel execution of complex operations across distributed data shards. The platform distinguishes itself through its specialized support for multi-tenant architectures and real-time analytical processing. It enables tenant-based
Automates the creation and removal of time-based table partitions using scheduled maintenance tasks to ensure continuous data ingestion and efficient retention.
Mac app that shows all open files, directories, sockets, pipes and devices in use by all running processes. Nice GUI for lsof.
Filters the displayed list by name, access mode, volume, type, location, or regular expression to narrow results.
pgloader is a command-line tool that automates the migration of data and schema from various source databases and file formats into PostgreSQL. It combines schema discovery, parallel data pipelines, and type casting into a single, declarative workflow, using PostgreSQL's COPY protocol for high-throughput bulk loading. The tool distinguishes itself by compiling a dedicated command language into concurrent reader-writer pipelines that handle schema introspection, data transformation, and error-resilient batch processing. It supports migrating entire databases from MySQL, MS SQL, SQLite, and Pos
Selectively includes or excludes tables from migration using name lists or regular expressions.
Implements spatial hashing and distance-based interest management to reduce network traffic in multiplayer games.
GreptimeDB is a distributed, open-source time-series database built for unified observability. It stores and queries metrics, logs, and traces together in a single columnar engine, supporting both SQL and PromQL for analysis. The database is designed as a Kubernetes-native operator with a decoupled compute and storage architecture, enabling horizontal scaling and multi-region deployment. What distinguishes GreptimeDB is its role as a multi-protocol ingestion gateway, accepting data through OpenTelemetry, Prometheus Remote Write, InfluxDB, Loki, Elasticsearch, Kafka, and MQTT protocols without
Lists tables in a database with optional name filtering for discovery.
CodeIgniter is a PHP web framework built on the Model-View-Controller pattern, designed for building full-stack web applications. It provides a lightweight toolkit with minimal configuration, organizing application logic into controllers, models, and views for clean separation of concerns. The framework includes a fluent query builder for constructing SQL statements programmatically, PSR-4 autoloading with namespace mapping, and a service-based dependency injection container for managing shared class instances. The framework distinguishes itself through its comprehensive set of built-in tools
Provides a utility method to enumerate all table names in the currently connected database.
Cortex is an open-source, horizontally scalable metrics platform that ingests, stores, and queries Prometheus-compatible time-series data with multi-tenant isolation. It accepts metrics via Prometheus remote write and OpenTelemetry, executes PromQL queries against both recent and historical data, and provides a Prometheus-compatible alerting and recording rule engine with an integrated Alertmanager. The system is built as a set of independently scalable microservices that use hash-ring-based sharding, gossip-based cluster membership, and tenant-aware object storage to distribute workloads acro
Divides timeseries across multiple sub-blocks using a hash of their labels for parallel compaction.
FastStream is an asynchronous Python framework designed for building event-driven microservices. It provides a unified abstraction layer for interacting with various message brokers, enabling developers to manage event production and consumption through a consistent interface while maintaining access to native provider-specific features. The framework centers on a decorator-based routing model that binds application logic directly to broker topics, supported by a built-in dependency injection container that resolves resources at runtime. The framework distinguishes itself through its deep int
Supports routing messages by partition keys to ensure data grouping and processing order.
This repository is a comprehensive collection of fully worked solutions to exercises and problems from the standard algorithms textbook by Cormen, Leiserson, Rivest, and Stein (CLRS). It serves as an educational reference for algorithm design and analysis, providing step-by-step reasoning, pseudocode, and mathematical proofs for a wide range of topics. The content spans core computer science areas: algorithm analysis with asymptotic notation, recurrence solving, and amortized cost analysis; data structure implementation and operations for binary search trees, red-black trees, B-trees, Fibonac
Illustrates the quicksort partition subroutine including pivot selection and array rearrangement.
Jitsu est une plateforme de données client conçue pour collecter, transformer et router les événements d'application vers des entrepôts de données et des outils marketing. Il fonctionne comme un moteur d'ingestion d'événements et un routeur d'entrepôt de données, capturant les données comportementales via des API et des SDK pour un traitement et un stockage en temps réel. La plateforme dispose d'un pipeline de données JavaScript programmable qui permet le filtrage, l'enrichissement et le remodelage des données d'événement pendant le transit. Elle inclut un outil de réconciliation d'identité client qui fusionne les identifiants d'utilisateurs anonymes et connus pour maintenir des profils clients persistants au sein d'un entrepôt. Le système couvre un large éventail de fonctionnalités, incluant la collecte d'événements multi-sources depuis des environnements web et mobiles, l'évolution automatique des schémas pour les entrepôts de destination et le routage multi-destination vers des plateformes SaaS et des bases de données SQL. Il fournit une suite d'outils développeur pour tester la logique de transformation et prend en charge le déploiement via Kubernetes ou des environnements Docker auto-hébergés.
Organizes records into time-based segments using a timestamp column to speed up analytical queries and reduce data scanned during lookups.
GAM is a command-line tool for administering Google Workspace and Cloud Identity. It translates command-line arguments into structured API calls, enabling administrators to manage users, groups, organizational units, and domain settings across a Google Workspace environment. The tool handles authentication through OAuth2 flows, service accounts, and workload identity federation, and supports multi-tenant configurations for managing multiple domains or cloud projects from a single installation. GAM distinguishes itself through its batch processing and automation capabilities. It can process la
Filters users, groups, files, or messages using regular expressions based on name or email.
This repository is a curated guide and implementation library of coding patterns used to solve data structures and algorithms problems. It serves as a technical interview study resource, providing a comprehensive set of strategies and computational logic examples for optimizing time and space complexity. The project focuses on standardized algorithmic patterns, including sliding windows, two pointers, and dynamic programming. It features specific implementations for a wide range of challenges, such as LeetCode problem solutions and specialized techniques like cyclic sort and bitwise XOR opera
Implements array partitioning that organizes three distinct values into separate zones in a single pass.
Screw est un outil d'analyse de schéma de base de données utilisé pour produire automatiquement de la documentation technique et des objets de données Java. Il scanne les tables de base de données pour exporter leurs définitions, transformant les métadonnées de schéma en rapports structurés et en code applicatif. Le projet génère des objets Java simples (POJO) basés sur les structures de tables existantes pour automatiser la modélisation des données. Il produit également une documentation de schéma de base de données aux formats HTML, Word et Markdown. Le système inclut des fonctionnalités d'exportation de tables avec filtrage par nom, l'extraction de schéma via JDBC, et la génération de code basée sur des templates pour mapper les types de colonnes de base de données aux types du langage Java.
Provides the ability to include or exclude specific database tables from the export process using name patterns.