opendatalab/MinerU
MinerU
MinerU is a document parsing pipeline designed to transform unstructured files into machine-readable, structured data. It utilizes deep learning models to perform layout analysis, identifying document regions and extracting complex content such as mathematical expressions. By combining these neural network inferences with geometric heuristics, the system reconstructs the reading order and structural hierarchy of documents to ensure accurate data representation.
The project distinguishes itself through a multi-stage processing workflow that integrates layout detection, optical character recognition, and formula extraction into a unified pipeline. It serializes all extracted features and spatial coordinates into a standardized format, ensuring that output remains consistent for downstream integration. To support verification, the tool includes a diagnostic suite that generates visual overlays, allowing users to inspect segmentation boundaries and reading order directly against the original source files.
The software provides a comprehensive framework for automated data extraction, organizing parsed elements into a page-based structure suitable for large-scale information retrieval. It is distributed as a Python-based package, with documentation and installation instructions available in the repository.
Features
- Deployment & Serving - Deploys deep learning models to classify content types and extract complex mathematical expressions from diverse visual inputs.
- Document Layout Analysis - Identifies document regions, tables, and text hierarchies to convert complex visual layouts into machine-readable data.
- Automated Data Extraction - Converts scanned or digital documents into structured data formats to enable large-scale information retrieval and analysis.
- Layout Reconstruction Algorithms - Applies geometric heuristics and spatial analysis to reassemble fragmented text blocks into a coherent reading order.
- Structured Data Extractors - Transforms unstructured document content into standardized, machine-readable formats for automated information retrieval.
- Document Processing Pipelines - Ingests unstructured files and normalizes them into structured data through a multi-stage deep learning pipeline.
- Multi-Stage Pipeline Processing - Orchestrates sequential document analysis tasks including layout detection, optical character recognition, and formula extraction.
- Data Processing - Tool for extracting high-quality content from PDFs and web pages.
- Data Processing Tools - Tool for high-quality extraction from PDFs and web pages.
- Document Parsing and Extraction - Converts PDF documents into structured Markdown and JSON.
- AI - Listed in the “AI 项目” section of the Great Open Source Project awesome list.
- Document Schema Normalizers - Organizes parsed document elements into a unified, page-based format to ensure consistent data structures for downstream applications.
- Visual Debugging Utilities - Generates visual overlays that highlight detected text segments and reading order to verify parsing accuracy.
- JSON-Schema - Encodes extracted document features and spatial coordinates into a standardized schema for seamless interoperability.
- Structured Data Exporters - Exports parsing results as structured JSON files to facilitate deeper data analysis through automated scripts.
