# eventual-inc/daft

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/eventual-inc-daft).**

5,225 stars · 401 forks · Rust · apache-2.0

## Links

- GitHub: https://github.com/Eventual-Inc/Daft
- Homepage: https://daft.ai
- awesome-repositories: https://awesome-repositories.com/repository/eventual-inc-daft.md

## Topics

`ai-engineering` `ai-pipeline` `arrow` `artificial-intelligence` `big-data` `data-engineering` `distributed` `distributed-computing` `distributed-systems` `embeddings` `etl` `huggingface` `iceberg` `machine-learning` `multimodal` `parquet` `python` `ray` `rust`

## Description

Daft is a distributed dataframe library and multimodal data processor designed to handle large-scale structured and unstructured data. It functions as a vectorized execution engine that processes tables alongside images, audio, and video, utilizing a unified schema to manage diverse data types.

The project distinguishes itself by combining distributed data engineering with large-scale AI inference. It provides an AI data pipeline for batch-optimizing model prompts and generating high-dimensional text embeddings, while utilizing zero-copy memory sharing to execute custom Python functions without processing overhead.

Its capabilities extend across cloud data lakehouse connectivity, supporting open table formats like Iceberg, Delta Lake, and Hudi. The engine employs lazy-evaluated execution plans and sampling-based schema inference to manage datasets that exceed single-node memory, scaling workloads from local cores to distributed Kubernetes clusters.

The system further includes a comprehensive suite for data transformation, covering columnar aggregation, window functions, and geospatial manipulation, as well as specialized tools for audio transcription and video frame extraction.

## Tags

### Artificial Intelligence & ML

- [Multimodal Processing](https://awesome-repositories.com/f/artificial-intelligence-ml/multimodal-processing.md) — Handles structured tables alongside unstructured media like images and audio within a single unified processing framework. ([source](https://cdn.jsdelivr.net/gh/eventual-inc/daft@main/README.md))
- [Batch Inference Pipelines](https://awesome-repositories.com/f/artificial-intelligence-ml/batch-inference-pipelines.md) — Provides a system for processing large-scale multimodal datasets through AI models via batch-optimized inference pipelines.
- [Inference Scaling](https://awesome-repositories.com/f/artificial-intelligence-ml/inference-scaling.md) — Runs batch prompts and generates embeddings across distributed GPU clusters to process massive datasets.
- [Multimodal Processing Engines](https://awesome-repositories.com/f/artificial-intelligence-ml/multimodal-input-processors/multimodal-processing-engines.md) — Ships a multimodal data processor that handles tables alongside images, audio, and video using vectorized execution.
- [AI Tool Execution](https://awesome-repositories.com/f/artificial-intelligence-ml/ai-tool-execution.md) — Executes model prompts and generates embeddings through optimized connections to external AI providers. ([source](https://cdn.jsdelivr.net/gh/eventual-inc/daft@main/README.md))
- [Audio Transcription](https://awesome-repositories.com/f/artificial-intelligence-ml/audio-transcription.md) — Converts audio files into textual segments with timestamps using speech-to-text models. ([source](https://docs.daft.ai/en/stable/examples/))
- [Synthetic Media Generators](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-ai-resources/diffusion-visual-models/synthetic-content-generators/synthetic-media-generators.md) — Generates synthetic images from textual prompts using local GPU-accelerated diffusion models. ([source](https://docs.daft.ai/en/stable/examples/))
- [AI Model Integrations](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-inference-serving/model-integration-pipelines/ai-model-integrations.md) — Provides interfaces for connecting multimodal data processing pipelines to various local and cloud-based AI models. ([source](https://docs.daft.ai/en/stable/install/))
- [Model Inference](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-inference-serving/model-integration-pipelines/model-inference.md) — Implements utilities for running model protocols, including text embedding, across multiple AI providers. ([source](https://docs.daft.ai/en/stable/api/))
- [Text Embedding Generators](https://awesome-repositories.com/f/artificial-intelligence-ml/text-embedding-generators.md) — Generates high-dimensional vector representations of text using GPU acceleration for vector database storage. ([source](https://docs.daft.ai/en/stable/examples/))
- [Embedding Generation Pipelines](https://awesome-repositories.com/f/artificial-intelligence-ml/vector-similarity-search/embedding-generation-pipelines.md) — Generates high-dimensional text embeddings and calculates vector similarity for storage in vector search engines.

### Data & Databases

- [Distributed Dataframes](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing/dataframe-processing/distributed-dataframes.md) — Provides a distributed dataframe library for processing large-scale structured and unstructured data across local cores or Kubernetes clusters.
- [Column Value Aggregations](https://awesome-repositories.com/f/data-databases/column-value-extraction/column-value-aggregations.md) — Calculates summary statistics like sums and averages across multiple columns for a single row. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Columnar Analytics](https://awesome-repositories.com/f/data-databases/columnar-analytics.md) — Utilizes vectorized columnar processing on contiguous memory blocks to maximize hardware utilization.
- [Data Ingestion](https://awesome-repositories.com/f/data-databases/data-ingestion.md) — Creates dataframes by loading and parsing data from in-memory sources, files, and external integrations. ([source](https://docs.daft.ai/en/stable/api/))
- [Distributed Computing](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing/distributed-processing-frameworks/distributed-computing.md) — Executes data processing tasks across a cluster of machines to handle datasets that exceed the memory of a single node. ([source](https://docs.daft.ai/en/stable/install/))
- [Python-Defined Transformations](https://awesome-repositories.com/f/data-databases/data-transformation-functions/python-defined-transformations.md) — Executes custom Python functions directly on data using zero-copy memory sharing for high-performance transformations. ([source](https://docs.daft.ai/))
- [Dataset Aggregations](https://awesome-repositories.com/f/data-databases/dataset-aggregations.md) — Performs functional aggregation and summary statistics across large distributed datasets. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Distributed Data Engines](https://awesome-repositories.com/f/data-databases/distributed-data-engines.md) — Executes complex transformations and aggregations on large datasets that exceed the memory of a single machine.
- [Grouped Aggregations](https://awesome-repositories.com/f/data-databases/grouped-aggregations.md) — Groups data by specific keys to calculate aggregate statistics like mean and count. ([source](https://docs.daft.ai/en/stable/api/))
- [Lazy Evaluation Frameworks](https://awesome-repositories.com/f/data-databases/lazy-evaluation-frameworks.md) — Implements a lazy-evaluated execution plan that defers data transformations until results are explicitly requested.
- [Multimodal Data Loading](https://awesome-repositories.com/f/data-databases/multimodal-data-storage/multimodal-data-loading.md) — Reads structured and unstructured data from cloud storage and AI repositories into a unified framework. ([source](https://docs.daft.ai/en/stable/quickstart/))
- [Cross-Language Zero-Copy Passings](https://awesome-repositories.com/f/data-databases/serialization-frameworks/zero-copy/cross-language-zero-copy-passings.md) — Employs zero-copy memory sharing to pass data between the core engine and Python functions without overhead.
- [Multimodal Unified Schemas](https://awesome-repositories.com/f/data-databases/structured-data-schemas/multimodal-unified-schemas.md) — Provides a unified schema that manages structured tables alongside images, audio, and video.
- [Schema Inference](https://awesome-repositories.com/f/data-databases/structured-data-schemas/schema-inference.md) — Automatically determines dataset structure through sampling without loading the entire file. ([source](https://docs.daft.ai/en/stable/api/))
- [Vectorized Execution Engines](https://awesome-repositories.com/f/data-databases/vectorized-execution-engines.md) — Implements a vectorized execution engine that optimizes memory usage and CPU efficiency for high-performance data transformations.
- [Data Deduplication Tools](https://awesome-repositories.com/f/data-databases/data-deduplication-tools.md) — Removes duplicate content from large text corpora using hashing algorithms. ([source](https://docs.daft.ai/en/stable/examples/))
- [Data Persistence and Storage](https://awesome-repositories.com/f/data-databases/data-engineering-infrastructure/data-persistence-storage.md) — Persists processed datasets to local or remote destinations including Parquet and S3. ([source](https://docs.daft.ai/en/stable/quickstart/))
- [Cloud Data Lake Integrations](https://awesome-repositories.com/f/data-databases/data-integration-synchronization/data-integration/cloud-data-lake-integrations.md) — Provides connectivity for reading and writing data using open table formats like Iceberg and Delta Lake.
- [Data Partitioning](https://awesome-repositories.com/f/data-databases/data-partitioning.md) — Divides large datasets into smaller segments using time-based or hash-based partitioning. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Data Processing](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing.md) — Provides general utilities for type casting, null filling, and conditional case-when expressions. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Data Source Connectivity Tools](https://awesome-repositories.com/f/data-databases/data-source-connectivity-tools.md) — Provides universal connectivity to data stored across cloud storage, table formats, and AI repositories. ([source](https://cdn.jsdelivr.net/gh/eventual-inc/daft@main/README.md))
- [Multi-Source Data Integration](https://awesome-repositories.com/f/data-databases/data-source-connectivity-tools/multi-source-data-integration.md) — Accesses data from diverse sources including cloud storage and enterprise catalogs without manual configuration. ([source](https://docs.daft.ai/))
- [Structured Types](https://awesome-repositories.com/f/data-databases/data-type-managers/structured-types.md) — Constructs structured data types from expressions and flattens nested fields into separate columns. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Date and Time Libraries](https://awesome-repositories.com/f/data-databases/date-and-time-libraries.md) — Performs temporal arithmetic and timezone conversions on timestamps. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Lakehouse Table Formats](https://awesome-repositories.com/f/data-databases/lakehouse-table-formats.md) — Reads and writes data using open table formats such as Iceberg, Delta Lake, and Hudi. ([source](https://docs.daft.ai/en/stable/install/))
- [Lazy Query Execution](https://awesome-repositories.com/f/data-databases/lazy-query-execution.md) — Defines data transformations and schemas using lazy evaluation to optimize the processing pipeline. ([source](https://docs.daft.ai/en/stable/quickstart/))
- [List Processing Tools](https://awesome-repositories.com/f/data-databases/list-processing-tools.md) — Provides operations to filter, sort, flatten, and map elements within list columns. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Numeric Calculators](https://awesome-repositories.com/f/data-databases/numeric-calculators.md) — Provides a suite of mathematical functions including trigonometry, logarithms, and rounding. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Remote Query Execution](https://awesome-repositories.com/f/data-databases/remote-query-execution.md) — Distributes processing tasks across a remote compute cluster to leverage external hardware resources. ([source](https://docs.daft.ai/en/stable/distributed/ray/))
- [Window Functions](https://awesome-repositories.com/f/data-databases/window-functions.md) — Implements context-aware window functions for complex calculations across sets of related rows. ([source](https://docs.daft.ai/en/stable/api/functions/))

### Development Tools & Productivity

- [User-Defined Data Functions](https://awesome-repositories.com/f/development-tools-productivity/custom-task-functions/sql-function-namespaces/function-definitions/user-defined-data-functions.md) — Allows the execution of custom user-defined logic directly on data stored within dataframes. ([source](https://docs.daft.ai/en/stable/api/))
- [Inference Batching](https://awesome-repositories.com/f/development-tools-productivity/batch-processing-pipelines/inference-batching.md) — Parallelizes model prompts across local processor cores to maximize throughput for large multimodal datasets. ([source](https://docs.daft.ai/en/stable/quickstart/))

### DevOps & Infrastructure

- [Distributed Task Orchestration](https://awesome-repositories.com/f/devops-infrastructure/distributed-task-orchestration.md) — Distributes data processing tasks across multiple machines to handle datasets that exceed single-node memory.
- [Distributed Data Workload Scaling](https://awesome-repositories.com/f/devops-infrastructure/worker-scaling/distributed-data-workload-scaling.md) — Transitions processing from local execution to distributed clusters via orchestration platforms. ([source](https://cdn.jsdelivr.net/gh/eventual-inc/daft@main/README.md))
- [Inference Capabilities](https://awesome-repositories.com/f/devops-infrastructure/ai-inference-infrastructure/inference-capabilities.md) — Enables text and image classification and embedding generation via external model providers. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Kubernetes Deployments](https://awesome-repositories.com/f/devops-infrastructure/kubernetes-deployments.md) — Runs data processing scripts on Kubernetes clusters using both single-node and distributed setups. ([source](https://docs.daft.ai/en/stable/distributed/kubernetes/))
- [Kubernetes Job Orchestration](https://awesome-repositories.com/f/devops-infrastructure/kubernetes-deployments/kubernetes-application-deployments/kubernetes-job-orchestration.md) — Deploys and scales data processing jobs on Kubernetes clusters to leverage remote compute.
- [Cloud Storage Connectors](https://awesome-repositories.com/f/devops-infrastructure/storage-configurations/storage-connection-apis/cloud-storage-connectors.md) — Interfaces with external storage providers and databases including S3 and various table formats. ([source](https://docs.daft.ai/en/stable/install/))

### Software Engineering & Architecture

- [Data Transformation Pipelines](https://awesome-repositories.com/f/software-engineering-architecture/data-transformation-pipelines.md) — Defines lazy-evaluated plans of operations for manipulating and computing data through multi-stage workflows. ([source](https://docs.daft.ai/en/stable/api/))
- [Resource Orchestration](https://awesome-repositories.com/f/software-engineering-architecture/performance-optimization-resources/memory-management/resource-orchestration.md) — Prevents out-of-memory errors using vectorized execution and intelligent resource management. ([source](https://docs.daft.ai/))

### Graphics & Multimedia

- [Audio Processing](https://awesome-repositories.com/f/graphics-multimedia/audio-music/audio-processing.md) — Extracts metadata and resamples audio files as part of a multimodal data processing pipeline. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Image Processing](https://awesome-repositories.com/f/graphics-multimedia/image-editing-processing/image-processing.md) — Decodes images and extracts metadata to generate perceptual hashes for duplicate detection. ([source](https://docs.daft.ai/en/stable/api/functions/))
- [Video File Processors](https://awesome-repositories.com/f/graphics-multimedia/media-processing-analysis/media-manipulation/media-processing/video-analysis-processing/video-file-processors.md) — Extracts metadata and captures specific frames from video files for analysis. ([source](https://docs.daft.ai/en/stable/api/functions/))

### Scientific & Mathematical Computing

- [Row Windowing](https://awesome-repositories.com/f/scientific-mathematical-computing/prefix-calculations/parallel-prefix-sum-calculators/rolling-aggregate-calculators/row-windowing.md) — Computes values across related rows to analyze local data trends within the dataframe. ([source](https://docs.daft.ai/en/stable/api/))