Eventual-Inc/Daft

Features

• Multimodal Processing - Handles structured tables alongside unstructured media like images and audio within a single unified processing framework.
• Distributed Dataframes - Provides a distributed dataframe library for processing large-scale structured and unstructured data across local cores or Kubernetes clusters.
• Batch Inference Pipelines - Provides a system for processing large-scale multimodal datasets through AI models via batch-optimized inference pipelines.
• Inference Scaling - Runs batch prompts and generates embeddings across distributed GPU clusters to process massive datasets.
• Multimodal Processing Engines - Ships a multimodal data processor that handles tables alongside images, audio, and video using vectorized execution.
• Column Value Aggregations - Calculates summary statistics like sums and averages across multiple columns for a single row.
• Columnar Analytics - Utilizes vectorized columnar processing on contiguous memory blocks to maximize hardware utilization.
• Data Ingestion - Creates dataframes by loading and parsing data from in-memory sources, files, and external integrations.
• Distributed Computing - Executes data processing tasks across a cluster of machines to handle datasets that exceed the memory of a single node.
• Python-Defined Transformations - Executes custom Python functions directly on data using zero-copy memory sharing for high-performance transformations.
• Dataset Aggregations - Performs functional aggregation and summary statistics across large distributed datasets.
• Distributed Data Engines - Executes complex transformations and aggregations on large datasets that exceed the memory of a single machine.
• Grouped Aggregations - Groups data by specific keys to calculate aggregate statistics like mean and count.
• Lazy Evaluation Frameworks - Implements a lazy-evaluated execution plan that defers data transformations until results are explicitly requested.
• Multimodal Data Loading - Reads structured and unstructured data from cloud storage and AI repositories into a unified framework.
• Cross-Language Zero-Copy Passings - Employs zero-copy memory sharing to pass data between the core engine and Python functions without overhead.
• Multimodal Unified Schemas - Provides a unified schema that manages structured tables alongside images, audio, and video.
• Schema Inference - Automatically determines dataset structure through sampling without loading the entire file.
• Vectorized Execution Engines - Implements a vectorized execution engine that optimizes memory usage and CPU efficiency for high-performance data transformations.
• User-Defined Data Functions - Allows the execution of custom user-defined logic directly on data stored within dataframes.
• Distributed Task Orchestration - Distributes data processing tasks across multiple machines to handle datasets that exceed single-node memory.
• Distributed Data Workload Scaling - Transitions processing from local execution to distributed clusters via orchestration platforms.
• Data Transformation Pipelines - Defines lazy-evaluated plans of operations for manipulating and computing data through multi-stage workflows.
• AI Tool Execution - Executes model prompts and generates embeddings through optimized connections to external AI providers.
• Audio Transcription - Converts audio files into textual segments with timestamps using speech-to-text models.
• Synthetic Media Generators - Generates synthetic images from textual prompts using local GPU-accelerated diffusion models.
• AI Model Integrations - Provides interfaces for connecting multimodal data processing pipelines to various local and cloud-based AI models.
• Model Inference - Implements utilities for running model protocols, including text embedding, across multiple AI providers.
• Text Embedding Generators - Generates high-dimensional vector representations of text using GPU acceleration for vector database storage.
• Embedding Generation Pipelines - Generates high-dimensional text embeddings and calculates vector similarity for storage in vector search engines.
• Data Deduplication Tools - Removes duplicate content from large text corpora using hashing algorithms.
• Data Persistence and Storage - Persists processed datasets to local or remote destinations including Parquet and S3.
• Cloud Data Lake Integrations - Provides connectivity for reading and writing data using open table formats like Iceberg and Delta Lake.
• Data Partitioning - Divides large datasets into smaller segments using time-based or hash-based partitioning.
• Data Processing - Provides general utilities for type casting, null filling, and conditional case-when expressions.
• Data Source Connectivity Tools - Provides universal connectivity to data stored across cloud storage, table formats, and AI repositories.
• Multi-Source Data Integration - Accesses data from diverse sources including cloud storage and enterprise catalogs without manual configuration.
• Structured Types - Constructs structured data types from expressions and flattens nested fields into separate columns.
• Date and Time Libraries - Performs temporal arithmetic and timezone conversions on timestamps.
• Lakehouse Table Formats - Reads and writes data using open table formats such as Iceberg, Delta Lake, and Hudi.
• Lazy Query Execution - Defines data transformations and schemas using lazy evaluation to optimize the processing pipeline.
• List Processing Tools - Provides operations to filter, sort, flatten, and map elements within list columns.
• Numeric Calculators - Provides a suite of mathematical functions including trigonometry, logarithms, and rounding.
• Remote Query Execution - Distributes processing tasks across a remote compute cluster to leverage external hardware resources.
• Window Functions - Implements context-aware window functions for complex calculations across sets of related rows.
• Inference Batching - Parallelizes model prompts across local processor cores to maximize throughput for large multimodal datasets.
• Inference Capabilities - Enables text and image classification and embedding generation via external model providers.
• Kubernetes Deployments - Runs data processing scripts on Kubernetes clusters using both single-node and distributed setups.
• Kubernetes Job Orchestration - Deploys and scales data processing jobs on Kubernetes clusters to leverage remote compute.
• Cloud Storage Connectors - Interfaces with external storage providers and databases including S3 and various table formats.
• Audio Processing - Extracts metadata and resamples audio files as part of a multimodal data processing pipeline.
• Image Processing - Decodes images and extracts metadata to generate perceptual hashes for duplicate detection.
• Video File Processors - Extracts metadata and captures specific frames from video files for analysis.
• Row Windowing - Computes values across related rows to analyze local data trends within the dataframe.
• Resource Orchestration - Prevents out-of-memory errors using vectorized execution and intelligent resource management.

Star history