24 repositorios
Optimized data storage structures for cache-efficient processing.
Distinguishing note: Focuses on low-level memory organization rather than high-level database management.
Explore 24 awesome GitHub repositories matching data & databases · Memory Layouts. Refine with filters or upvote what's useful.
Pandas is a high-performance data analysis library that provides a comprehensive framework for manipulating, cleaning, and transforming structured datasets. It centers on labeled one-dimensional and two-dimensional data structures, allowing users to construct, filter, and reshape tabular information while performing complex arithmetic and logical operations. The library distinguishes itself through a sophisticated indexing engine that enables automatic data alignment during calculations and relational merges. By utilizing a block-based memory layout, it optimizes cache locality for vectorized
Provides contiguous memory block storage to optimize cache locality and vectorized operations.
NumPy is a foundational library for scientific computing in Python, providing a comprehensive framework for managing and manipulating large-scale numerical information. It centers on high-performance multidimensional array objects that serve as the primary data structure for complex mathematical operations and data analysis workflows. The library distinguishes itself through specialized mechanisms for handling multidimensional data, including advanced indexing, slicing, and broadcasting techniques that allow for efficient operations across arrays of varying shapes. It utilizes strided metadat
Uses strided metadata to enable efficient, zero-copy slicing of multidimensional arrays.
This project is a machine learning array framework and tensor computation library designed for high-performance numerical computing. It provides a comprehensive suite of tools for constructing and training neural networks, featuring an automatic differentiation engine that facilitates gradient-based optimization and complex mathematical modeling. The library distinguishes itself through a unified memory architecture that allows data to be shared across CPU and GPU devices without explicit copies, significantly reducing data movement overhead. Its execution model relies on a lazy evaluation en
Forces arrays into row-major memory layouts to ensure alignment and compatibility.
This project is a cross-platform graphics and compute framework that provides a unified, hardware-agnostic abstraction layer for rendering and parallel processing. It enables developers to build high-performance applications that execute consistently across diverse operating systems and hardware backends, including Vulkan, Metal, and DirectX. By mapping high-level graphics commands to native APIs, it serves as a portable foundation for both real-time 3D rendering and general-purpose GPU computing. The framework distinguishes itself through a robust architecture that supports both native deskt
Configures alignment and padding of vertex, index, and uniform data structures for hardware-specific memory requirements.
Arrow is a cross-language development platform for in-memory data. It provides a standardized, language-independent columnar memory format designed to accelerate analytical operations and improve memory efficiency on modern computing hardware. By utilizing a schema-driven approach, the framework enables the efficient organization of both flat and nested data structures. The project functions as an analytical data processing engine that facilitates high-performance computation directly on memory-resident datasets. It distinguishes itself through a zero-copy architecture, which allows multiple
Organizes data in contiguous memory blocks to maximize CPU cache efficiency and enable vectorized processing.
SciPy is a scientific computing library for Python that provides a comprehensive collection of mathematical algorithms and numerical tools for research and engineering. It functions as a high-performance numerical analysis framework, bridging high-level Python code with compiled C and Fortran routines to execute complex computations at hardware speeds. The library is built upon array-based data structures that utilize strided memory layouts to enable efficient data manipulation and slicing. By employing vectorized operation dispatch and linking to optimized hardware-specific linear algebra li
Utilizes strided memory layouts to enable efficient slicing and manipulation of multidimensional data without copying.
This project is a curated collection of programming exercises designed to build proficiency in numerical computing and data manipulation. It provides a structured learning path for mastering multidimensional array operations, vectorized arithmetic, and statistical analysis. The repository focuses on developing practical expertise in array-based workflows, emphasizing techniques such as memory management, efficient data processing, and the replacement of explicit loops with vectorized operations. Users engage with hands-on challenges that cover the full lifecycle of numerical data, from initia
Organizes data into user-defined fields within contiguous memory blocks to represent complex records.
Odin is a compiled, statically typed systems programming language designed for high-performance software development. It focuses on pragmatic low-level memory control, providing a toolset for manual memory management and precise control over hardware utilization. The language is distinguished by its flexible memory model, which includes custom allocators and precise data layout capabilities to optimize resource usage. It features a comprehensive foreign function interface for importing assembly files and linking with external libraries using configurable calling conventions. The type system
Enables efficient data packing using bit fields and specific record layouts for cache-efficient processing.
cuDF is a GPU-accelerated dataframe library and data processing engine designed for manipulating and analyzing large tabular datasets. It provides a high-level API for executing filtering, joining, and aggregating operations directly on GPU hardware. The project integrates the Apache Arrow memory format to enable zero-copy data transfers and includes a just-in-time compiler for executing custom user-defined functions on the GPU. The library features specialized acceleration for existing workflows by redirecting standard Pandas dataframe calls and Polars query plans to a GPU backend. It also p
Implements optimized columnar memory layouts to maximize GPU bandwidth and SIMD execution efficiency.
Torch7 is a scientific computing environment and tensor computation library used for deep learning research and numerical analysis. It functions as a Lua-based framework for training neural networks and learning agents, providing a toolkit for implementing architectures and training through reinforcement learning algorithms. The project is distinguished by its tight integration with C, utilizing a binding layer to map high-level scripting to low-level C structures for direct memory access. It supports hardware-accelerated computation by offloading linear algebra and convolution operations to
Implements strided memory layouts to manipulate tensor dimensions and shapes without duplicating data buffers.
Vaex is a high-performance Apache Arrow DataFrame library and out-of-core data processing engine designed to handle billion-row tabular datasets in Python. It functions as a lazy evaluation framework that defers computations and transformations until results are required, enabling the processing of datasets that exceed available system RAM by mapping files directly from disk. The project distinguishes itself as a tool for big data visualization and exploration, specifically integrated for use within interactive notebooks. It provides specialized capabilities for machine learning feature engin
Implements an Apache Arrow columnar memory layout to enable high-speed data access and efficient interoperability.
FastImageCache is an iOS image caching library that provides a persistent disk-based image store. It utilizes a persistent bitmap cache to store images in uncompressed formats and incorporates an image pre-processing pipeline to optimize assets before they are committed to storage. The library optimizes rendering performance by using memory-mapped image tables for constant-time retrieval and byte-aligned data layouts to prevent memory copies. It organizes images of identical dimensions into shared tables and manages disk space through a least-recently-used cache eviction system. The project
Optimizes rendering performance by aligning image rows to memory boundaries to prevent expensive memory copies.
go-tools is a collection of utilities for Go static analysis and memory layout optimization. It provides a toolset designed to analyze source code to detect bugs and dead code, alongside specialized tools for optimizing how structs are arranged in memory. The project includes a memory alignment visualizer to display physical memory layouts and padding, as well as a struct layout optimizer that reorders fields to minimize memory padding. Additionally, it provides a boilerplate generator to automate the creation of registration and test files required for developing custom Go analyzers. The to
Provides a utility for inspecting memory alignment, field sizes, and padding of Go structs.
Provides native handling of interleaved, planar, and custom memory layouts without data copying.
Este proyecto es una colección integral de librerías y toolkits de C++ que proporcionan implementaciones de referencia para estructuras de datos, algoritmos de grafos y lógica de bits. Sirve como una referencia de algoritmos en C++ que contiene más de 180 problemas de programación resueltos y un toolkit especializado para programación competitiva. El repositorio se distingue por sus extensas librerías de manipulación de bits de bajo nivel para comprobaciones de paridad, detección de endianness y lógica basada en XOR. También proporciona una amplia gama de soluciones de referencia para desafíos algorítmicos complejos que involucran backtracking, teoría de grafos y programación dinámica. La superficie de capacidades cubre organizadores de datos lineales y jerárquicos fundamentales, incluyendo listas enlazadas, pilas, colas y árboles de búsqueda binaria. Incluye un conjunto completo de algoritmos de grafos para búsqueda de caminos y árboles de expansión, varios métodos de ordenamiento y búsqueda, transformaciones de matrices y utilidades de procesamiento de cadenas. Además, cubre funciones computacionales matemáticas, compresión de datos sin pérdida y cifrados criptográficos básicos.
Implements byte order reversal to convert data between big-endian and little-endian representations.
Magnum es una suite de middleware de C++ para el desarrollo de gráficos multiplataforma y visualización de datos en tiempo real. Proporciona una capa de renderizado agnóstica al hardware que traduce comandos gráficos en llamadas específicas de la plataforma, garantizando un comportamiento consistente en diferentes controladores de GPU y API como Vulkan. El proyecto se centra en desacoplar la lógica de la aplicación del hardware subyacente a través de gráficos abstractos y utilidades del sistema. Cuenta con un importador de recursos basado en plugins para activos 3D y audio, un grafo de escena jerárquico para transformaciones espaciales y un sistema de eventos basado en señales de alto rendimiento para la comunicación. Las capacidades amplias incluyen álgebra lineal y matemáticas vectoriales, procesamiento de geometría de malla y la gestión de contextos de GPU. El kit de herramientas también cubre la reproducción de audio espacial, la integración de hardware de VR y optimizaciones de memoria de bajo nivel como diseños escalonados y asignaciones alineadas. La biblioteca puede integrarse en proyectos principales como un subproyecto de CMake.
Organizes interleaved data into contiguous blocks and strided views to enhance data locality and SIMD efficiency.
c3c is the compiler for the C3 programming language, transforming source code into executable binaries, static libraries, or dynamic libraries using an LLVM backend. It implements a system based on result-based error handling, scoped memory pooling, and a semantic macro system. The compiler provides first-class support for hardware-backed SIMD vectors that map directly to processor instructions and enables runtime polymorphism through interface-based dynamic dispatch. The project covers a broad set of low-level capabilities, including manual and pooled memory management, inline assembly inte
Controls bit-level data layout for big-endian and little-endian storage and overlapping ranges.
nalgebra es una biblioteca de álgebra lineal para Rust que proporciona operaciones de matriz y vector con soporte para dimensiones tanto en tiempo de compilación como en tiempo de ejecución. Funciona como una biblioteca de análisis numérico y una biblioteca de matrices dispersas, ofreciendo un framework matemático capaz de ejecutarse en entornos embebidos y WebAssembly sin requerir la biblioteca estándar de Rust. El proyecto se distingue como una biblioteca de transformación geométrica, utilizando coordenadas homogéneas, cuaterniones e isometrías para manejar rotaciones, traslaciones y proyecciones 3D. Implementa una variedad de descomposiciones de matrices —incluyendo LU, QR, Cholesky, SVD y descomposición de valores propios— para resolver sistemas lineales y analizar matrices. La biblioteca cubre áreas de capacidad amplias, incluyendo computación geométrica para transformaciones espaciales, utilidades de gráficos por computadora para la composición de matrices de proyección y exportación de datos de shaders, y gestión especializada de matrices dispersas utilizando almacenamiento comprimido por filas y columnas. También proporciona herramientas de gestión de datos para la inicialización, redimensionamiento y análisis de archivos Matrix Market.
Provides non-owning references to matrix sub-sections using memory offsets to avoid data copying.
ndarray es una biblioteca de arreglos multidimensionales para Rust que sirve como framework de álgebra lineal y herramienta de computación científica. Proporciona la infraestructura central para crear y manipular arreglos de n-dimensiones, funcionando tanto como un procesador de arreglos paralelo como un kit de herramientas para el análisis de datos numéricos. La biblioteca se distingue por proporcionar cortes (slicing) y vistas de memoria eficientes, lo que permite compartir datos sin copiarlos. Aprovecha bibliotecas matemáticas de backend optimizadas para la multiplicación de matrices de alta velocidad y distribuye iteraciones matemáticas pesadas a través de múltiples hilos de CPU para acelerar el procesamiento. El proyecto cubre una amplia gama de operaciones matemáticas, incluyendo aritmética elemento a elemento, agregación de datos basada en ejes y cálculos de producto punto. También incluye utilidades integrales para la manipulación de arreglos como el cambio de forma, aplanamiento, apilamiento y generación de cuadrículas de coordenadas, junto con soporte para la generación de arreglos aleatorios y serialización.
Maps multidimensional indices to flat memory buffers using axis-specific step sizes for efficient zero-copy slicing.
NumCpp es un framework de C++ y biblioteca de computación numérica que proporciona un kit de herramientas para la gestión de arrays multidimensionales y rutinas matemáticas. Funciona como una implementación en C++ del ecosistema NumPy, ofreciendo un framework de computación científica para gestionar tensores y realizar ecuaciones algebraicas complejas. El proyecto permite la manipulación de arrays de alto rendimiento dentro de un entorno C++ sin depender de un runtime de Python. Se distingue por proporcionar una interfaz similar a NumPy para ejecutar álgebra lineal, gestionar estructuras de datos multidimensionales y realizar procesamiento numérico. La biblioteca cubre una amplia gama de capacidades, incluyendo operaciones algebraicas de matrices, gestión de geometría de arrays mediante slicing y reshaping, y la generación de distribuciones aleatorias. También incluye herramientas para el análisis de datasets, estadísticas de arrays y la importación/exportación de datos numéricos mediante formatos binarios y de texto.
Utilizes strided memory mapping to allow efficient array reshaping and slicing without copying data.