ibis-project/ibis
Ibis
Ibis is a portable Python dataframe library and multi-backend query engine that provides a unified interface for executing data transformations across diverse compute engines. It functions as a Python SQL expression compiler and dialect transpiler, allowing users to define data logic once and execute it across cloud warehouses, embedded databases, and distributed clusters without rewriting code.
The project distinguishes itself through a database backend abstraction that decouples transformation logic from the underlying execution engine. It enables polyglot data workflows by mixing raw SQL strings with programmatic Python expressions and provides a dataframe interop layer to convert query results between various memory formats.
Its capability surface covers a broad range of data analysis operations, including row filtering, aggregation, and geospatial processing, as well as comprehensive schema and database hierarchy management. The system also supports data streaming via a unified batch and streaming API, as well as integration with cloud storage and various file formats for data import and export.
Features
- Portable Data Expression APIs - Provides a unified programmatic interface for defining data transformations that work across different backend engines.
- Multi-Engine Execution Backends - Executes data transformations across different SQL databases and cloud warehouses using a single unified Python interface.
- Backend Adapter Development - Provides a framework for creating database adapters that expose a unified dataframe API across different engines.
- Backend Connection Management - Manages connectivity settings, credentials, and authentication for diverse data store backends.
- Column Selection Utilities - Provides utilities for selecting specific columns using names, objects, or pattern-based selectors.
- Column Value Aggregations - Computes summary statistics like mean, max, min, and sum across columns or groups.
- Derived Column Generation - Creates new columns from existing data or constant literal values.
- Database Engine Abstractions - Provides a unified interface that decouples transformation logic from specific database engines.
- Conditional Row Filters - Isolates specific data subsets by selecting rows based on one or more boolean conditions.
- Dataframe Format Conversion - Implements a bridge to convert query execution results between SQL backends and formats like Pandas, Polars, and PyArrow.
- Dataframe Interoperability - Transforms query results into common data structures such as Pandas, Polars, and PyArrow.
- Lazy Expression Execution - Executes deferred table expressions against a backend to produce in-memory data objects.
- Lazy Operation Chaining - Allows combining multiple data manipulation steps into a single lazy pipeline for optimized execution.
- Polyglot Query Workflows - Allows mixing raw SQL strings with programmatic Python expressions within a single data pipeline.
- Dialect Transformers - Rewrites SQL constructs between different database dialects to ensure cross-backend compatibility.
- Tabular Data Sorting - Arranges rows in ascending or descending order based on one or more column values.
- SQL Dialect Compilation - Translates chains of Python data operations into specific database SQL dialects.
- Intermediate Representation Translation - Uses an abstract intermediate format to decouple data transformation logic from the final target query language.
- Query Intermediate Representations - Maintains an internal abstract model of data transformations that can be transpiled into multiple target query languages.
- Query Execution Plans - Builds a chain of operations as a lazy expression tree that only materializes results upon explicit request.
- Pluggable Execution Backends - Allows the runtime to switch between local in-memory engines and remote distributed clusters via connection strings.
- Analysis Ecosystem Integrations - Integrates query results directly with external visualization, dashboarding, and machine learning libraries.
- Change Data Capture Streams - Enables subscribing to tables or views to receive continuous change data capture updates.
- Type-Based Selection - Enables the inclusion or exclusion of groups of columns based on their data type.
- Compiled SQL Inspection - Allows users to inspect the exact SQL query compiled from a Python dataframe expression.
- Raw SQL Execution - Allows the execution of arbitrary SQL strings for administrative tasks and table creation.
- In-Memory DataFrame Imports - Imports data from in-memory Python structures like pandas DataFrames or PyArrow Tables into database tables.
- Unified Batch and Stream Processing Engines - Offers a single interface for managing both traditional batch data processing and real-time streaming workloads.
- In-Memory Format Adapters - Adapts query execution results into in-memory dataframe formats like Pandas or PyArrow.
- Database Views - Generates virtual tables from transformation expressions and stores them within the backend.
- External File Imports - Registers external CSV, JSON, Parquet, or Delta Lake files as tables within the database.
- File-Based Data Exports - Writes the results of data expressions directly to structured file formats such as CSV, JSON, Parquet, and Delta Lake.
- SQL Geospatial Analysis - Executes specialized spatial queries and processes geospatial data using SQL-based analysis.
- Result Streaming - Returns query results as an iterator of dataframes or record batches for memory-efficient processing.
- Hybrid Object-SQL Execution - Enables mixing raw SQL strings with programmatic dataframe operations within a single pipeline.
- Row Groupings - Provides the ability to create row groupings based on shared values in one or more columns for collective analysis.
- Runtime Backend Switching - Allows changing the underlying execution engine at runtime without rewriting data transformation logic.
- SQL to Expression Wrapping - Transforms raw SQL queries into structured lazy table expressions for further programmatic manipulation.
- Upsert Operations - Implements atomic operations to either insert new records or update existing ones based on a key.
- Cloud Storage Integrations - Enables reading and writing data from cloud blob storage using integrated secret management.
- Data Analysis - Portable dataframe library for multiple backends.
- Data Analysis and Processing - Portable Python dataframe library.
- Database Clients - Portable Python dataframe library.
