Awesome GitHub RepositoriesHigh-Performance Data Analysis

dlib is a C++ machine learning toolkit and data analysis framework. It provides a collection of algorithms and utilities for building predictive modeling applications and performing statistical analysis on large datasets within native C++ environments. The project functions as a binding library that wraps low-level C++ machine learning algorithms into high-level Python scripting interfaces. This allows for the integration of high-performance native implementations with Python for machine learning development. The framework covers the implementation of predictive models, the execution of mach

Apache Druid is a real-time OLAP database and distributed analytics engine. It functions as a columnar time-series database designed for high-performance analytical queries and the real-time ingestion of streaming and batch datasets. The system provides a framework for high-concurrency analytics, allowing multiple simultaneous users to execute SQL and native queries across large-scale data. It supports mixed data ingestion, combining real-time streaming and batch loading into a single system for unified analysis. The platform includes capabilities for distributed cluster management, enabling

Apache Druid is a real-time analytics database and distributed columnar time-series store designed for sub-second analytical queries. It functions as a data platform featuring a distributed SQL query engine and a real-time data ingestion system for moving historical and streaming data from external sources. The system is distinguished by its ability to provide low-latency analytics under high concurrency to power operational dashboards. It implements a Kerberos-secured environment for user authentication and employs a shared-nothing cluster architecture to enable horizontal scaling. The plat

cuDF is a GPU-accelerated dataframe library and data processing engine designed for manipulating and analyzing large tabular datasets. It provides a high-level API for executing filtering, joining, and aggregating operations directly on GPU hardware. The project integrates the Apache Arrow memory format to enable zero-copy data transfers and includes a just-in-time compiler for executing custom user-defined functions on the GPU. The library features specialized acceleration for existing workflows by redirecting standard Pandas dataframe calls and Polars query plans to a GPU backend. It also p

RisingWave is a cloud-native streaming database and real-time analytics engine that uses standard SQL to process continuous data streams. It functions as a streaming data lakehouse, combining the capabilities of a streaming SQL database with a platform that integrates streaming ingestion with open table formats. The system is distinguished by its use of the PostgreSQL wire protocol, allowing it to integrate with existing SQL tools and drivers. It employs a decoupled compute and storage architecture, persisting streaming state and materialized views in cloud object storage to enable independen

Vaex is a high-performance Apache Arrow DataFrame library and out-of-core data processing engine designed to handle billion-row tabular datasets in Python. It functions as a lazy evaluation framework that defers computations and transformations until results are required, enabling the processing of datasets that exceed available system RAM by mapping files directly from disk. The project distinguishes itself as a tool for big data visualization and exploration, specifically integrated for use within interactive notebooks. It provides specialized capabilities for machine learning feature engin

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

Highway es una biblioteca de C++ portátil y capa de abstracción de hardware diseñada para escribir código SIMD (Single Instruction, Multiple Data). Proporciona una interfaz unificada que mapea la lógica de datos paralelos a varios conjuntos de instrucciones de CPU, permitiendo el desarrollo de software de alto rendimiento que se ejecuta en diferentes arquitecturas de procesador sin requerir ensamblador específico de la arquitectura. El proyecto cuenta con un despachador de instrucciones dinámico que selecciona el conjunto de instrucciones de CPU más eficiente en tiempo de ejecución basado en el hardware detectado. También admite la especialización de destino estática y mecanismos extensibles para añadir nuevos destinos de hardware u operaciones SIMD personalizadas. La biblioteca cubre una amplia gama de operaciones vectoriales, incluyendo aritmética elemento a elemento, reducción de carriles (lane reduction), barajado (shuffling) y ejecución condicional enmascarada. Incluye una biblioteca matemática vectorizada, un gestor de memoria para asignación alineada y operaciones de carga-almacenamiento enmascaradas, y primitivas para criptografía acelerada por hardware. Se proporcionan herramientas para la compilación y validación automatizada de instrucciones aceleradas por hardware en múltiples arquitecturas de procesador.

This project is a high-performance tabular data processing framework for R, designed to handle massive datasets with memory efficiency and speed. It provides an enhanced data structure that utilizes reference semantics and in-place modification to perform complex transformations without the overhead of unnecessary object copying. The library distinguishes itself through its low-level architectural optimizations, including multi-threaded parallel processing, radix-based sorting, and memory-mapped file parsing. By offloading critical data manipulation and aggregation routines to compiled C code

Este proyecto es una librería estándar impulsada por la comunidad para el lenguaje de programación Fortran, proporcionando una colección completa de algoritmos, estructuras de datos y utilidades de sistema. Está diseñada para extender las capacidades nativas del lenguaje, ofreciendo un kit de herramientas unificado para computación científica, análisis numérico y programación de propósito general. La librería se distingue por una arquitectura modular que utiliza despacho de interfaces genéricas y especialización en tiempo de compilación para asegurar un alto rendimiento a través de varios tipos de datos. Proporciona abstracciones estandarizadas para backends numéricos externos, permitiendo a los desarrolladores cambiar entre implementaciones de referencia internas y librerías optimizadas de proveedores. El código base está organizado en namespaces jerárquicos para prevenir colisiones de símbolos y soporta gestión de memoria estática para mantener un rendimiento predecible en entornos de alto rendimiento. La librería cubre una amplia superficie de capacidades, incluyendo operaciones de álgebra lineal, cálculo de funciones matemáticas y generación de distribuciones aleatorias. También proporciona infraestructura esencial para procesamiento de texto, interacción con el sistema de archivos y gestión del entorno. Para apoyar el ciclo de vida de desarrollo, el proyecto incluye herramientas integradas para pruebas unitarias, validación basada en aserciones y registro de diagnósticos en tiempo de ejecución. El proyecto sigue patrones convencionales de sistemas de compilación para asegurar la compatibilidad con gestores de paquetes modernos y entornos de desarrollo externos.