24 repository-uri
Optimized data storage structures for cache-efficient processing.
Distinguishing note: Focuses on low-level memory organization rather than high-level database management.
Explore 24 awesome GitHub repositories matching data & databases · Memory Layouts. Refine with filters or upvote what's useful.
Pandas is a high-performance data analysis library that provides a comprehensive framework for manipulating, cleaning, and transforming structured datasets. It centers on labeled one-dimensional and two-dimensional data structures, allowing users to construct, filter, and reshape tabular information while performing complex arithmetic and logical operations. The library distinguishes itself through a sophisticated indexing engine that enables automatic data alignment during calculations and relational merges. By utilizing a block-based memory layout, it optimizes cache locality for vectorized
Provides contiguous memory block storage to optimize cache locality and vectorized operations.
NumPy is a foundational library for scientific computing in Python, providing a comprehensive framework for managing and manipulating large-scale numerical information. It centers on high-performance multidimensional array objects that serve as the primary data structure for complex mathematical operations and data analysis workflows. The library distinguishes itself through specialized mechanisms for handling multidimensional data, including advanced indexing, slicing, and broadcasting techniques that allow for efficient operations across arrays of varying shapes. It utilizes strided metadat
Uses strided metadata to enable efficient, zero-copy slicing of multidimensional arrays.
This project is a machine learning array framework and tensor computation library designed for high-performance numerical computing. It provides a comprehensive suite of tools for constructing and training neural networks, featuring an automatic differentiation engine that facilitates gradient-based optimization and complex mathematical modeling. The library distinguishes itself through a unified memory architecture that allows data to be shared across CPU and GPU devices without explicit copies, significantly reducing data movement overhead. Its execution model relies on a lazy evaluation en
Forces arrays into row-major memory layouts to ensure alignment and compatibility.
This project is a cross-platform graphics and compute framework that provides a unified, hardware-agnostic abstraction layer for rendering and parallel processing. It enables developers to build high-performance applications that execute consistently across diverse operating systems and hardware backends, including Vulkan, Metal, and DirectX. By mapping high-level graphics commands to native APIs, it serves as a portable foundation for both real-time 3D rendering and general-purpose GPU computing. The framework distinguishes itself through a robust architecture that supports both native deskt
Configures alignment and padding of vertex, index, and uniform data structures for hardware-specific memory requirements.
Arrow is a cross-language development platform for in-memory data. It provides a standardized, language-independent columnar memory format designed to accelerate analytical operations and improve memory efficiency on modern computing hardware. By utilizing a schema-driven approach, the framework enables the efficient organization of both flat and nested data structures. The project functions as an analytical data processing engine that facilitates high-performance computation directly on memory-resident datasets. It distinguishes itself through a zero-copy architecture, which allows multiple
Organizes data in contiguous memory blocks to maximize CPU cache efficiency and enable vectorized processing.
SciPy is a scientific computing library for Python that provides a comprehensive collection of mathematical algorithms and numerical tools for research and engineering. It functions as a high-performance numerical analysis framework, bridging high-level Python code with compiled C and Fortran routines to execute complex computations at hardware speeds. The library is built upon array-based data structures that utilize strided memory layouts to enable efficient data manipulation and slicing. By employing vectorized operation dispatch and linking to optimized hardware-specific linear algebra li
Utilizes strided memory layouts to enable efficient slicing and manipulation of multidimensional data without copying.
This project is a curated collection of programming exercises designed to build proficiency in numerical computing and data manipulation. It provides a structured learning path for mastering multidimensional array operations, vectorized arithmetic, and statistical analysis. The repository focuses on developing practical expertise in array-based workflows, emphasizing techniques such as memory management, efficient data processing, and the replacement of explicit loops with vectorized operations. Users engage with hands-on challenges that cover the full lifecycle of numerical data, from initia
Organizes data into user-defined fields within contiguous memory blocks to represent complex records.
Odin is a compiled, statically typed systems programming language designed for high-performance software development. It focuses on pragmatic low-level memory control, providing a toolset for manual memory management and precise control over hardware utilization. The language is distinguished by its flexible memory model, which includes custom allocators and precise data layout capabilities to optimize resource usage. It features a comprehensive foreign function interface for importing assembly files and linking with external libraries using configurable calling conventions. The type system
Enables efficient data packing using bit fields and specific record layouts for cache-efficient processing.
cuDF is a GPU-accelerated dataframe library and data processing engine designed for manipulating and analyzing large tabular datasets. It provides a high-level API for executing filtering, joining, and aggregating operations directly on GPU hardware. The project integrates the Apache Arrow memory format to enable zero-copy data transfers and includes a just-in-time compiler for executing custom user-defined functions on the GPU. The library features specialized acceleration for existing workflows by redirecting standard Pandas dataframe calls and Polars query plans to a GPU backend. It also p
Implements optimized columnar memory layouts to maximize GPU bandwidth and SIMD execution efficiency.
Torch7 is a scientific computing environment and tensor computation library used for deep learning research and numerical analysis. It functions as a Lua-based framework for training neural networks and learning agents, providing a toolkit for implementing architectures and training through reinforcement learning algorithms. The project is distinguished by its tight integration with C, utilizing a binding layer to map high-level scripting to low-level C structures for direct memory access. It supports hardware-accelerated computation by offloading linear algebra and convolution operations to
Implements strided memory layouts to manipulate tensor dimensions and shapes without duplicating data buffers.
Vaex is a high-performance Apache Arrow DataFrame library and out-of-core data processing engine designed to handle billion-row tabular datasets in Python. It functions as a lazy evaluation framework that defers computations and transformations until results are required, enabling the processing of datasets that exceed available system RAM by mapping files directly from disk. The project distinguishes itself as a tool for big data visualization and exploration, specifically integrated for use within interactive notebooks. It provides specialized capabilities for machine learning feature engin
Implements an Apache Arrow columnar memory layout to enable high-speed data access and efficient interoperability.
FastImageCache is an iOS image caching library that provides a persistent disk-based image store. It utilizes a persistent bitmap cache to store images in uncompressed formats and incorporates an image pre-processing pipeline to optimize assets before they are committed to storage. The library optimizes rendering performance by using memory-mapped image tables for constant-time retrieval and byte-aligned data layouts to prevent memory copies. It organizes images of identical dimensions into shared tables and manages disk space through a least-recently-used cache eviction system. The project
Optimizes rendering performance by aligning image rows to memory boundaries to prevent expensive memory copies.
go-tools is a collection of utilities for Go static analysis and memory layout optimization. It provides a toolset designed to analyze source code to detect bugs and dead code, alongside specialized tools for optimizing how structs are arranged in memory. The project includes a memory alignment visualizer to display physical memory layouts and padding, as well as a struct layout optimizer that reorders fields to minimize memory padding. Additionally, it provides a boilerplate generator to automate the creation of registration and test files required for developing custom Go analyzers. The to
Provides a utility for inspecting memory alignment, field sizes, and padding of Go structs.
Provides native handling of interleaved, planar, and custom memory layouts without data copying.
Acest proiect este o colecție cuprinzătoare de biblioteci și toolkit-uri C++ care oferă implementări de referință pentru structuri de date, algoritmi pe grafuri și logică pe biți. Acesta servește drept referință de algoritmi C++ conținând peste 180 de probleme de programare rezolvate și un toolkit specializat pentru programarea competitivă. Repository-ul se distinge prin biblioteci extinse de manipulare a biților la nivel scăzut pentru verificări de paritate, detectarea endianness-ului și logică bazată pe XOR. De asemenea, oferă o gamă largă de soluții de referință pentru provocări algoritmice complexe care implică backtracking, teoria grafurilor și programare dinamică. Suprafața de capabilități acoperă organizatori de date liniari și ierarhici fundamentali, inclusiv liste înlănțuite, stive, cozi și arbori binari de căutare. Include o suită completă de algoritmi pe grafuri pentru pathfinding și arbori de acoperire minimă, diverse metode de sortare și căutare, transformări de matrice și utilitare pentru procesarea șirurilor de caractere. În plus, acoperă funcții matematice computaționale, compresia datelor fără pierderi și cifruri criptografice de bază.
Implements byte order reversal to convert data between big-endian and little-endian representations.
Magnum este o suită de middleware C++ pentru dezvoltarea grafică cross-platform și vizualizarea datelor în timp real. Oferă un strat de randare hardware-agnostic care traduce comenzile grafice în apeluri specifice platformei, asigurând un comportament consistent între diferite drivere GPU și API-uri precum Vulkan. Proiectul se concentrează pe decuplarea logicii aplicației de hardware-ul subiacent prin utilitare grafice și de sistem abstracte. Dispune de un importator de resurse bazat pe plugin-uri pentru active 3D și audio, un graf de scenă ierarhic pentru transformări spațiale și un sistem de evenimente bazat pe semnale de înaltă performanță pentru comunicare. Capabilitățile largi includ algebra liniară și matematica vectorială, procesarea geometriei mesh și gestionarea contextelor GPU. Toolkit-ul acoperă, de asemenea, redarea audio spațială, integrarea hardware-ului VR și optimizări de memorie de nivel scăzut, cum ar fi layout-urile strided și alocările aliniate. Biblioteca poate fi integrată în proiectele părinte ca un subproiect CMake.
Organizes interleaved data into contiguous blocks and strided views to enhance data locality and SIMD efficiency.
c3c is the compiler for the C3 programming language, transforming source code into executable binaries, static libraries, or dynamic libraries using an LLVM backend. It implements a system based on result-based error handling, scoped memory pooling, and a semantic macro system. The compiler provides first-class support for hardware-backed SIMD vectors that map directly to processor instructions and enables runtime polymorphism through interface-based dynamic dispatch. The project covers a broad set of low-level capabilities, including manual and pooled memory management, inline assembly inte
Controls bit-level data layout for big-endian and little-endian storage and overlapping ranges.
nalgebra este o bibliotecă de algebră liniară pentru Rust care oferă operații cu matrice și vectori cu suport pentru dimensiuni atât la compilare, cât și la runtime. Funcționează ca o bibliotecă de analiză numerică și o bibliotecă de matrice rare (sparse matrix), oferind un framework matematic capabil să ruleze în medii embedded și WebAssembly fără a necesita biblioteca standard Rust. Proiectul se distinge ca o bibliotecă de transformări geometrice, utilizând coordonate omogene, cuaternioni și izometrii pentru a gestiona rotații 3D, translații și proiecții. Implementează o varietate de descompuneri de matrice — inclusiv LU, QR, Cholesky, SVD și eigendecomposition — pentru a rezolva sisteme liniare și a analiza matrice. Biblioteca acoperă arii largi de capabilități, inclusiv calcul geometric pentru transformări spațiale, utilitare de computer graphics pentru compunerea matricelor de proiecție și exportul datelor pentru shadere, precum și gestionarea specializată a matricelor rare folosind stocarea comprimată pe rânduri și coloane. Oferă, de asemenea, instrumente de gestionare a datelor pentru inițializarea matricelor, redimensionare și parsarea fișierelor Matrix Market.
Provides non-owning references to matrix sub-sections using memory offsets to avoid data copying.
ndarray is a multidimensional array library for Rust that serves as a linear algebra framework and scientific computing tool. It provides the core infrastructure for creating and manipulating n-dimensional arrays, functioning as both a parallel array processor and a toolkit for numerical data analysis. The library distinguishes itself by providing efficient slicing and memory views, allowing for data sharing without copying. It leverages optimized backend math libraries for high-speed matrix multiplication and distributes heavy mathematical iterations across multiple CPU threads to accelerate
Maps multidimensional indices to flat memory buffers using axis-specific step sizes for efficient zero-copy slicing.
NumCpp is a C++ framework and numerical computing library that provides a toolkit for multi-dimensional array management and mathematical routines. It functions as a C++ implementation of the NumPy ecosystem, offering a scientific computing framework for managing tensors and performing complex algebraic equations. The project enables high-performance array manipulation within a C++ environment without relying on a Python runtime. It distinguishes itself by providing a NumPy-like interface for executing linear algebra, managing multi-dimensional data structures, and performing numerical proces
Utilizes strided memory mapping to allow efficient array reshaping and slicing without copying data.