Awesome GitHub RepositoriesColumnar Data Processors

Polars is a high-performance columnar data processing library designed for efficient analytical workflows. It functions as a structured data library that organizes information into typed columns, utilizing the Apache Arrow memory format to enable zero-copy data sharing and cache-friendly, vectorized operations. The engine is built to handle large-scale tabular datasets, providing both local and distributed analytical runtimes that scale from single-machine environments to multi-node clusters. The project distinguishes itself through a sophisticated lazy query engine that constructs abstract e

Arrow is a cross-language development platform for in-memory data. It provides a standardized, language-independent columnar memory format designed to accelerate analytical operations and improve memory efficiency on modern computing hardware. By utilizing a schema-driven approach, the framework enables the efficient organization of both flat and nested data structures. The project functions as an analytical data processing engine that facilitates high-performance computation directly on memory-resident datasets. It distinguishes itself through a zero-copy architecture, which allows multiple

Perspective is a columnar data analytics engine and high-performance visualization component powered by WebAssembly. It provides a system for analyzing and visualizing large or streaming datasets through interactive data grids and charts, utilizing a compiled binary to achieve near-native performance within the browser. The project distinguishes itself through a WebSocket-based data streaming interface and deep Apache Arrow integration, which minimize memory overhead when synchronizing tables between servers and clients. It acts as a remote query proxy capable of translating visualization con

Perspective is a columnar data analytics library and streaming data visualization engine. It provides an interactive data grid component and notebook analytics widgets designed for processing high-volume data and rendering interactive charts and grids. The system utilizes a high-performance query engine to enable real-time data analysis and streaming dataset visualization. It supports the creation of customizable dashboards and reports that update automatically as new data arrives without requiring full dataset reloads. The project covers large-scale dataset analytics through a schema-driven

Fory is a cross-language serialization framework and binary data serializer designed to convert complex object graphs into a compact binary format for high-performance data exchange. It includes an IDL-based schema compiler to transform interface definition language files into type-safe native data models and a schema evolution manager to maintain forward and backward compatibility. The project features a zero-copy data access layer that allows reading specific fields from binary rows without deserializing the entire object. It supports dual-mode serialization, enabling a toggle between a por

Velox est un moteur d'exécution de requêtes C++ haute performance et une bibliothèque de traitement de données colonnaires. Il sert de framework composable pour implémenter des moteurs de requêtes analytiques, fournissant un évaluateur d'expressions vectorisées et une boîte à outils pour les systèmes de gestion de données. Le projet se distingue par son utilisation de l'exécution colonnaire vectorisée et de l'allocation mémoire basée sur des arènes pour traiter des jeux de données à grande échelle. Il propose des optimisations spécialisées telles que la mise en cache des tables de jointure broadcast, le push-down de filtres dynamiques et l'encodage par dictionnaire pour réduire la surcharge mémoire et accélérer les lectures analytiques. Le moteur couvre un large éventail de capacités analytiques, incluant l'implémentation de jointures hash, merge et semi, ainsi que l'agrégation parallèle multi-étapes et le calcul de fonctions de fenêtre. Il fournit des primitives pour le stockage colonnaire en mémoire, le décodage de données Parquet et l'intégration avec le stockage cloud. L'extensibilité est assurée par un système d'enregistrement de fonctions pour des fonctions scalaires et d'agrégation personnalisées, avec des bindings de haut niveau disponibles pour connecter la logique C++ à Python.

Kùzu is an embedded property graph database engine designed for high-performance analytical queries and local data management. It operates as a library within the host application process, utilizing a columnar-based storage architecture and just-in-time query compilation to execute complex graph traversals and pattern matching efficiently. By mapping database files directly into system memory, it ensures data durability and high-speed access while maintaining ACID-compliant transactional integrity. The engine distinguishes itself by integrating vector similarity search and full-text search di