Projects sharing features with Narwhals

DuckLake is an integrated data lake and catalog format

Turning PySpark Into a Universal DataFrame API

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types. The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions witho

| Tutorials | API Documentation | Chat with us on slack! | | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |…

Extension for Scikit-learn is a seamless way to speed up your Scikit-learn application

A package which efficiently applies any function to a pandas dataframe or series in the fastest available manner

QuackOSM: an open-source Python and CLI tool for reading OpenStreetMap PBF files using DuckDB

This project is a machine learning array framework and tensor computation library designed for high-performance numerical computing. It provides a comprehensive suite of tools for constructing and training neural networks, featuring an automatic differentiation engine that facilitates gradient-based optimization and complex mathematical modeling. The library distinguishes itself through a unified memory architecture that allows data to be shared across CPU and GPU devices without explicit copies, significantly reducing data movement overhead. Its execution model relies on a lazy evaluation en

Modin is a distributed dataframe library and parallel data processing engine designed to handle large datasets that exceed system memory. It functions as a distributed computing framework that parallelizes data manipulation tasks across multiple CPU cores or clusters to increase throughput and avoid memory errors. The project mirrors the Pandas API, allowing for the distribution of data workflows without changing core code logic. It utilizes a pluggable backend interface, which enables users to switch between different distributed execution engines to optimize performance based on available h

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

A lightweight Snowflake emulator built with Go and DuckDB for local development and testing

Polars is a high-performance columnar data processing library designed for efficient analytical workflows. It functions as a structured data library that organizes information into typed columns, utilizing the Apache Arrow memory format to enable zero-copy data sharing and cache-friendly, vectorized operations. The engine is built to handle large-scale tabular datasets, providing both local and distributed analytical runtimes that scale from single-machine environments to multi-node clusters. The project distinguishes itself through a sophisticated lazy query engine that constructs abstract e

Xarray is a Python multidimensional array library and labeled dataset framework. It extends the NumPy data structure by adding labels to arrays, allowing for the organization of complex N-dimensional data using named dimensions and coordinates. The library provides a NetCDF data interface for reading and writing scientific data formats such as NetCDF and Zarr. It enables scientific array computing by maintaining the relationship between data and physical coordinates during mathematical operations. The project covers multidimensional data analysis, geospatial data manipulation, and climate da

missingno is a Python library for the visualization and analysis of missing data patterns. It provides a set of tools to profile dataset completeness, map data gaps, and quantify the volume of null values across variables. The library differentiates itself through a nullity correlation analyzer and a hierarchical data clustering tool. These components allow for the detection of systemic dependencies and trends by measuring how the absence of one variable relates to the absence of another. The toolset covers broader data quality auditing and exploratory analysis capabilities. It includes feat

Utilities for using duckdb with react.

Pandas integration with sklearn

A pure Java stand-alone SQL Transpiler, Column- and Lineage Resolver for translating various large RDBMS SQL Dialects into a few smaller RDBMS Dialects for Unit Testing. Based on JSQLParser.

DuckDB implementation of Unleasharp.DB.Base. This repository provides a DuckDB-specific implementation that leverages the base abstraction layer for common database operations.

Vaex is a high-performance Apache Arrow DataFrame library and out-of-core data processing engine designed to handle billion-row tabular datasets in Python. It functions as a lazy evaluation framework that defers computations and transformations until results are required, enabling the processing of datasets that exceed available system RAM by mapping files directly from disk. The project distinguishes itself as a tool for big data visualization and exploration, specifically integrated for use within interactive notebooks. It provides specialized capabilities for machine learning feature engin

Ydata-profiling is an automated exploratory data analysis framework designed to generate comprehensive statistical reports and visual summaries from dataframes. It functions as a diagnostic tool for assessing data quality, identifying missing values, duplicates, and outliers, while providing a scalable engine for profiling massive datasets across distributed enterprise environments. The project distinguishes itself through its ability to handle large-scale data through distributed task orchestration and lazy stream processing, which minimizes memory overhead during complex computations. It in