katanaml/sparrow
Sparrow
Sparrow est une plateforme d'extraction de documents par LLM et un moteur d'inférence basé sur la vision, conçu pour convertir des images et des PDF en données structurées validées. Il fonctionne comme un orchestrateur de workflow agentique qui enchaîne des tâches de classification, d'extraction et de validation dans des pipelines multi-étapes.
Le système se distingue par une couche d'inférence agnostique au backend qui gère les modèles sur des GPU locaux, Apple Silicon et des fournisseurs cloud. Il utilise le "visual grounding" basé sur les coordonnées pour mapper le texte extrait à des boîtes englobantes précises et utilise un guidage par indices pour orienter l'attention et normaliser les formats de données.
La plateforme couvre les workflows d'intelligence documentaire, incluant le traitement spécialisé d'images de tableaux pour maintenir l'intégrité structurelle et une validation basée sur des schémas pour vérifier l'exactitude des champs extraits. Elle fournit également un tableau de bord d'analyse documentaire pour surveiller les performances de l'API, les statistiques d'utilisation et l'état du système.
L'architecture inclut un système d'extension par plugins pour intégrer des bibliothèques tierces utilisées dans l'indexation et l'orchestration.
Features
- Intelligent Document Processing - Provides a platform for intelligent document processing, combining classification, extraction, and validation into multi-step pipelines.
- Vision-Language Model Backends - Uses vision-capable language models to parse document layouts and convert visual content into structured data.
- Agentic Workflow Pipelines - Implements automated pipelines that chain LLM instructions and external tools for complex document analysis and error recovery.
- Hardware Acceleration Backends - Manages extraction pipelines across diverse hardware accelerators including local and cloud-based backends.
- Image Text Extractions - Recognizes and extracts text and key-value pairs from images and PDFs as structured data.
- Vision-Language Orchestrators - Manages model inference across local GPUs, Apple Silicon, and cloud providers to process visual document data.
- Hardware-Agnostic Inference Layers - Provides a hardware-agnostic inference layer that routes processing to local GPUs, Apple Silicon, or cloud providers.
- Structured Document Extraction - Converts visual document layouts into machine-readable structured formats using vision-capable models.
- Vision-Language Inference - Implements a vision-language inference engine that executes multimodal models across various hardware backends.
- Workflow Orchestration - Combines document classification and data extraction into a single AI workflow pipeline with visual monitoring.
- Document Field Validations - Checks extracted document fields against schemas to verify the presence and correctness of required data.
- Structured Data Extraction - Parses complex tables and text from documents into predefined schemas with bounding box coordinate mapping.
- PDF Coordinate Extraction - Extracts precise bounding box coordinates for recognized text regions within PDF pages.
- Agentic Workflow Orchestrators - Functions as an orchestrator that chains LLM classification, extraction, and validation tasks with integrated error recovery.
- Pipeline Orchestrators - Chains classification, extraction, and validation tasks into sequenced pipelines with error recovery.
- Schema-Driven Validations - Verifies extracted document fields against predefined structural definitions to ensure data correctness.
- Extraction Coordinate Annotations - Generates bounding box coordinates for extracted elements to provide visual grounding for the data.
- Visual Coordinate Mapping - Maps extracted text to precise bounding box coordinates for visual grounding within documents.
- Tabular Data Extraction - Extracts complex tabular data from documents while maintaining structural integrity through specialized vision processing.
- OCR Document Conversion - Converts images and PDFs into validated structured data using vision models and schema-based validation.
- Local Model Backends - Supports running inference across a variety of backends including local GPUs, Apple Silicon, and cloud providers.
- Attention Steering Hints - Uses hint-based configuration files to steer model attention and normalize extracted data formats.
- Extraction Hinting - Uses hint-based model steering to guide attention and normalize data formats during the extraction process.
- Document Processing Pipelines - Extracts and analyzes data across documents containing multiple pages using orchestrated pipelines.
- Table Structure Detections - Identifies tabular grids and merges cells within document layouts to crop them for specialized inference.
- Image-Based Table Extractors - Identifies tabular regions and crops them into images for specialized inference to preserve structural integrity.
- Document Table Extractors - Maps large or multi-column tables from documents to structured schemas using an intermediate processing pipeline.
- Document Analysis Dashboards - Provides a visual dashboard for monitoring API performance, usage analytics, and the operational health of extraction pipelines.
- Data Processing - Solution for efficient data extraction from documents and images.
- Data Processing Tools - Solution for efficient data extraction from documents and images.
