ucbepic/docetl

Features

• Document and Unstructured Extraction - Transforms large collections of unstructured documents into structured, queryable tables using language models.
• Parallel Map-Reduce Tools - Coordinates data processing through parallel map, reduce, and filter operations to transform unstructured text into structured tables.
• Structured Data Extraction - Transforms large collections of unstructured documents into queryable tables using schema-constrained LLM extraction.
• Entity Clustering and Canonicalization - Canonicalizes duplicate entities across multiple documents to ensure data consistency through clustering and synthesis.
• LLM-Based Data Transformations - Applies prompt-based transformations to input items with automatic retries and strict schema validation.
• LLM Orchestration - Designs and executes complex map-reduce workflows that integrate language model transformations with deterministic Python code.
• Schema Definition - Allows specifying the structure of extracted data using basic types and nested objects to ensure consistent formats.
• LLM Schema Outputs - Uses predefined structural types and nested objects to constrain language model outputs into consistent formats.
• Declarative Workflow Definitions - Provides a way to define the end-to-end flow and transformation logic of document processing pipelines via configuration files.
• LLM-Integrated Extraction Pipelines - Orchestrates workflows that chain document ingestion and layout analysis with model-based structured data generation.
• ETL Workflows - Implements automated ETL workflows to extract, clean, and transform data from documents into predefined schemas.
• Declarative Pipeline Construction - Implements a declarative interface for defining complex data operations and workflows to transform unstructured datasets into tables.
• Iterative Information Aggregation - Condenses information from multiple documents into structured summaries through iterative aggregation steps.
• Semantic Similarity Joins - Merges disparate datasets by calculating embedding-based similarity scores when exact primary keys are unavailable.
• PDF and HTML Content Extraction - Provides capabilities to extract text and metadata from PDF documents into standardized objects for further processing.
• Agent Tool Execution - Allows the execution of external tools or functions during transformations to perform real-time lookups and actions.
• Automated Data Validation - Evaluates the accuracy and operational cost of extraction pipelines through iterative refinement and schema validation.
• Recursive Document Synthesis - Groups unstructured documents into hierarchical clusters to generate synthesized summaries across varying granularities.
• Document-Based - Creates high-quality datasets by aggregating, clustering, and resolving entities across collections of unstructured files.
• Document Clustering Frameworks - Groups unstructured documents into hierarchical clusters based on semantic similarity to generate synthesized summaries.
• Document Summarization - Condenses key information from multiple documents into structured summaries using a reduction process.
• Prompt Optimizers - Refines model prompts and examples to improve the accuracy and reliability of the extraction pipeline.
• Multi-Document Syntheses - Aggregates and synthesizes information across multiple documents grouped by a common key into unified outputs.
• Natural Language Entity Extraction - Uses natural language instructions and map-reduce operators to extract structured metrics and entities from unstructured text.
• Workflow Performance Optimizations - Optimizes workflow efficiency by swapping models, rewriting prompts, and replacing expensive LLM tasks with deterministic code.
• LLM-Powered Document Transformations - Processes individual documents with language models to perform data extraction, content filtering, and semantic ranking.
• PDF Text Extraction - Provides the capability to download and pull text from publicly accessible PDF URLs for processing.
• LLM Output Constraints - Enforces specific requirements and logic-based constraints on data extracted from large language models.
• Dataset Loading - Imports data from JSON or CSV files and maps fields to variables for use in processing operations.
• Entity Resolution - A feature that resolves variations of the same entity into a single canonical name using embedding-based blocking and comparison.
• Multi-Format Data Loading - Imports data from standard files or custom parsing tools for non-standard formats like audio and PDFs.
• Prompt Playgrounds - Ships an interactive environment for refining prompts and testing model parameters in real-time.
• Extraction Pipeline Prototyping - Provides a real-time interface for developing and refining document processing workflows to test transformations instantly.
• Natural Language Pipeline Generators - Provides a natural language interface to generate data processing workflows for extracting information from unstructured files.
• Deterministic Pipeline Step Execution - Supports the execution of custom Python scripts within pipelines for computational processing or external library integration.
• Pipeline Accuracy Optimizers - Includes an automated process to determine the best operational settings and thresholds to improve data extraction accuracy.
• Iterative Refinement Loops - Implements deterministic checks and refinement loops to correct errors and ensure LLM outputs comply with defined schemas.
• Data Ingestion and Parsing - System for agentic data processing and ETL workflows.

Star history