Projects sharing features with Vaex

Pandas is a high-performance data analysis library that provides a comprehensive framework for manipulating, cleaning, and transforming structured datasets. It centers on labeled one-dimensional and two-dimensional data structures, allowing users to construct, filter, and reshape tabular information while performing complex arithmetic and logical operations. The library distinguishes itself through a sophisticated indexing engine that enables automatic data alignment during calculations and relational merges. By utilizing a block-based memory layout, it optimizes cache locality for vectorized

DataFrame is a C++ tabular data library and manipulation engine designed for managing heterogeneous data in contiguous memory. It functions as a statistical analysis framework and time series analysis toolkit, providing the means to store, index, and transform multidimensional datasets. The project distinguishes itself through a high-performance execution model that utilizes column-major storage, SIMD-aligned memory allocation, and a thread-pool for parallel computations. It employs a visitor-based algorithm dispatch system and policy-driven transformations to decouple data processing logic f

This project is a high-performance tabular data processing framework for R, designed to handle massive datasets with memory efficiency and speed. It provides an enhanced data structure that utilizes reference semantics and in-place modification to perform complex transformations without the overhead of unnecessary object copying. The library distinguishes itself through its low-level architectural optimizations, including multi-threaded parallel processing, radix-based sorting, and memory-mapped file parsing. By offloading critical data manipulation and aggregation routines to compiled C code

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

cuDF is a GPU-accelerated dataframe library and data processing engine designed for manipulating and analyzing large tabular datasets. It provides a high-level API for executing filtering, joining, and aggregating operations directly on GPU hardware. The project integrates the Apache Arrow memory format to enable zero-copy data transfers and includes a just-in-time compiler for executing custom user-defined functions on the GPU. The library features specialized acceleration for existing workflows by redirecting standard Pandas dataframe calls and Polars query plans to a GPU backend. It also p

NumPy is a foundational library for scientific computing in Python, providing a comprehensive framework for managing and manipulating large-scale numerical information. It centers on high-performance multidimensional array objects that serve as the primary data structure for complex mathematical operations and data analysis workflows. The library distinguishes itself through specialized mechanisms for handling multidimensional data, including advanced indexing, slicing, and broadcasting techniques that allow for efficient operations across arrays of varying shapes. It utilizes strided metadat

VisiData is a terminal-based interactive data analysis tool and browser designed for exploring, filtering, and sorting large tabular datasets. It functions as a structured data inspector that loads and flattens complex formats like JSON, XML, and PCAP into interactive sheets, as well as a terminal file manager for navigating directories and performing staged filesystem operations. The project distinguishes itself by rendering data visualizations, such as scatter plots and histograms, directly in the terminal using Unicode Braille characters. It provides a Python-based data wrangling environme

r4ds is a data science curriculum and educational resource designed for mastering the R programming language. It provides a structured learning path for the end-to-end process of importing, tidying, transforming, and visualizing data. The project emphasizes a reproducible data science guide and a comprehensive curriculum for data wrangling. It includes specialized tutorials on the grammar of graphics for layered data visualization and technical publications created with Quarto that blend executable code with narrative prose. The material covers a broad range of analytical capabilities, inclu

This project is a Python data science curriculum and programming tutorial collection. It provides a structured set of educational notebooks and scripts designed to teach data analysis, machine learning, and deep learning. The repository serves as a learning path for building and tuning predictive models, including regression, decision trees, and neural networks. It includes a data visualization guide for creating financial time-series plots and a multiprocessing reference for implementing parallel task execution and shared memory synchronization. The curriculum covers broader capability area

This is an interactive notebook-based course that teaches machine learning from Python fundamentals through deep learning and natural language processing. It uses real datasets and multiple frameworks within a structured, hands-on curriculum that combines concise explanations with executable code cells, built-in datasets, and embedded exercise checkpoints. Learning progresses through data preparation and exploration, classical machine learning workflows, computer vision with convolutional neural networks, and natural language processing with deep learning, all delivered as a cohesive progressi

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

Xarray is a Python multidimensional array library and labeled dataset framework. It extends the NumPy data structure by adding labels to arrays, allowing for the organization of complex N-dimensional data using named dimensions and coordinates. The library provides a NetCDF data interface for reading and writing scientific data formats such as NetCDF and Zarr. It enables scientific array computing by maintaining the relationship between data and physical coordinates during mathematical operations. The project covers multidimensional data analysis, geospatial data manipulation, and climate da

Pandas integration with sklearn

Scikit-learn is a machine learning library for predictive data analysis that provides a collection of algorithms for supervised and unsupervised learning. It functions as a comprehensive toolkit for data preprocessing, dimensionality reduction, and model selection, allowing users to classify data objects, predict continuous values, and cluster similar items based on historical patterns. The project is defined by a unified interface design where objects either learn from data, transform data, or chain these operations into sequential workflows. To ensure performance on large or high-dimensiona

Remeda is a type-safe functional utility library for TypeScript designed for building data transformation pipelines. It provides a toolkit of helper functions for manipulating arrays and objects while maintaining strict type integrity throughout the process. The library is characterized by its support for both data-first and data-last calling styles. It utilizes lazy evaluation to process data collections, evaluating transformations only when the final result is requested to avoid creating intermediate collection copies. The toolkit covers collection manipulation, function composition, and t

Chronicle Queue is a high-performance data handling system featuring off-heap message queues, memory-mapped file stores, and replicated message stores. It provides a binary compatible memory layout that enables different programming languages to share data without serialization overhead. The system utilizes a replicated message store to synchronize data across multiple nodes, ensuring high availability and instant failover. Its memory-mapped architecture supports deterministic replay from disk and low-latency data recording. The project implements off-heap memory management and zero-allocati

This project is a Python education repository and programming tutorial designed to teach language fundamentals, from basic syntax and variables to advanced concepts. It serves as a data science starter kit and a guide for REST API integration. The repository provides instructional scripts and sample code covering object-oriented programming patterns and asynchronous programming. It includes practical demonstrations for fetching and processing JSON data from external web services using HTTP requests. The materials cover a broad capability surface including data analysis workflows with interac

This project is an automated machine learning framework and toolkit designed for training and tuning custom models for classification, regression, and recommendations. It functions as a multimodal machine learning toolkit capable of processing and training models using a combination of text, image, audio, and sensor data. The framework distinguishes itself as a multimodal data processor that can handle and visualize large datasets on a single machine using column-oriented disk storage. It includes a core machine learning model generator that converts trained models into formats compatible wit

Joblib is a suite of utilities for parallelizing computational workloads and optimizing the storage of large numerical datasets and function results. It functions as a parallel computing library and multiprocessing wrapper that distributes function execution across multiple CPU cores to accelerate independent tasks and computational loops. The project provides a disk caching framework that persists expensive function outputs to the filesystem, re-evaluating them only when input arguments change. It further specializes in the serialization of large numerical arrays, utilizing efficient compres

tsfresh is an automated feature engineering tool and library designed to extract statistical characteristics from raw time series data. It transforms sequential data into tabular datasets, converting time series into a flat format where each row represents a unique entity and columns represent extracted features. The project distinguishes itself through a parallel data processing framework that distributes heavy computational workloads across multiple CPU cores. It also implements hypothesis-based feature selection to identify the most predictive characteristics and filter out irrelevant ones

Danfo.js is a data analysis and preprocessing library for JavaScript that provides high-performance labeled data structures. It implements data frames and series to enable complex data analysis, statistical computing, and the manipulation of structured tabular data. The project serves as a machine learning preprocessing library, offering utilities for categorical label encoding, one-hot encoding, and numeric feature scaling and standardization. It specifically facilitates the conversion of labeled data structures into tensors for model training and evaluation. The library covers a broad set

Featuretools is a Python data science library and automated feature engineering framework designed to create predictive features from multiple related datasets. It automates the data preparation and transformation steps required for machine learning models through deep feature synthesis. The library enables the automatic generation of comprehensive feature tables by applying recursive transformations to relational data. It supports the transformation of unstructured text into structured numeric features and allows users to define custom primitives to extend the synthesis process with specific

This project is an educational resource providing practical code examples and implementations of machine learning algorithms using the Python language. It serves as a guide for constructing predictive pipelines, clustering models, and dimensionality reduction within the Scikit-Learn ecosystem. The repository includes comprehensive demonstrations for supervised and unsupervised learning, as well as detailed examples for implementing neural networks and deep architectures. It also provides practical guidance on exporting model parameters to JSON and wrapping trained models in web APIs for produ

This project is an educational resource and a collection of instructional materials for performing data manipulation and statistical analysis using Python. It provides a comprehensive set of guides and code examples for using the Pandas, NumPy, and Matplotlib libraries to analyze structured data. The resource includes a dedicated guide for reshaping, cleaning, and aggregating tabular data and time series via Pandas, alongside a reference for high-performance vectorized operations and linear algebra using NumPy. It also features tutorials for creating publication-quality charts, distribution p

dlt is a Python data ingestion tool and ETL pipeline framework designed to fetch data from diverse sources and persist it into structured destinations. It functions as a schema inference engine that automatically detects data types and flattens nested JSON structures into relational tables, moving data from sources to lakehouses, warehouses, or vector databases. The project distinguishes itself through AI-powered pipeline generation, using large language models to scaffold extraction code and connectors for REST APIs. It also supports multimodal vector storage and specialized population of ve

Kùzu is an embedded property graph database engine designed for high-performance analytical queries and local data management. It operates as a library within the host application process, utilizing a columnar-based storage architecture and just-in-time query compilation to execute complex graph traversals and pattern matching efficiently. By mapping database files directly into system memory, it ensures data durability and high-speed access while maintaining ACID-compliant transactional integrity. The engine distinguishes itself by integrating vector similarity search and full-text search di