bytedance/Dolphin
Dolphin
Dolphin is a multimodal layout analyzer and image-to-structure converter that transforms photographed or digital document images into machine-readable structured data. It functions as an LLM document parser, utilizing vision-language models to simultaneously predict spatial layout and text content.
The system is designed as a concurrent document processor, employing parallel document parsing to process multiple elements across distributed compute nodes. This high-throughput approach reduces the total time required to convert large volumes of images into structured formats.
The project covers spatial document layout mapping to identify bounding boxes and generate natural reading order sequences. It provides capabilities for granular content retrieval, allowing for the targeted extraction of specific document elements such as tables, formulas, and code blocks through prompt-based parsing.
Features
- Vision-Language Inference - Uses vision-language inference to simultaneously predict spatial layout and text content from document images.
- Document Layout Analyzers - Provides a multimodal layout analyzer that identifies spatial arrangements and reading orders of text, tables, and figures in images.
- Pixel Coordinate Mappings - Maps high-level bounding boxes and regions to exact pixel coordinates for document layout identification.
- Image-to-Text Transformers - Uses transformer-based mapping to convert image pixels directly into structured text sequences.
- Content Parsing Prompts - Uses targeted text instructions to guide the model in isolating specific data types like tables or formulas.
- Document Image Transformations - Converts raw pixel data from photographs or digital scans into machine-readable structured text formats.
- Document Layout Analysis - Parses and extracts structural information from complex documents to identify text, tables, and layout hierarchies.
- Structured Document Extraction - Converts visual document layouts into machine-readable formats like JSON or Markdown.
- Parallel Processing - Implements parallel document parsing across distributed nodes to reduce the total time required for large-volume image conversion.
- Vision-Based Document Parsers - Uses multimodal vision models to interpret document layouts and convert them into structured text.
- Reading Order Predictors - Analyzes text and spatial layout to determine the logical reading sequence of document elements.
- Reading Order Reconstruction - Determines the natural reading order by linking disparate layout elements into a coherent structural sequence.
- Complex Document Extraction - Identifies and parses specific components such as tables, formulas, and paragraphs from complex document sources.
- Parallel Processing - Employs concurrent execution of document transformation tasks to improve overall processing throughput.
- Concurrent Data Processors - Distributes computational workloads across multiple cores to accelerate the conversion of images into structured data.
- Document Processing Engines - Provides high-performance pipelines for converting large volumes of images into structured data through parallel execution.
- Granular Content Extractions - Parses specific content types, such as tables or code blocks, from images for detailed data retrieval.
- Multimodal Models - Multimodal model for image and text integration.
- Data Processing - Document image parsing using heterogeneous anchor prompting.
- Data Processing Tools - Document image parsing using heterogeneous anchor prompting.
- Document Parsing and Extraction - Official repository for document image parsing via heterogeneous prompting.
