zjunlp/DeepKE
DeepKE
DeepKE ist ein Toolkit und Framework zur Wissensextraktion, das darauf ausgelegt ist, unstrukturierte Texte in strukturierte Wissensgraphen zu transformieren. Es bietet eine Pipeline zur Identifizierung und Klassifizierung benannter Entitäten, semantischer Beziehungen und Ereignisse und konvertiert rohe Datensätze in strukturierte Tripel.
Das Projekt nutzt Large Language Models als Tool-Caller durch ein standardisiertes Kontextprotokoll, um automatisierte Datenextraktionsprozesse voranzutreiben. Es unterstützt schema-gesteuerte Extraktion über mehrere Domänen und zweisprachige Texte hinweg und verwendet gemeinsame Entitäts- und Beziehungsextraktion, um Komponenten in einer einzigen strukturierten Ausgabe zu identifizieren.
Das Toolkit umfasst Funktionen für Modelltraining und Fine-Tuning, Hyperparameter-Optimierung und Datenvorbereitung via Distant Supervision und automatisierter Beziehungslabeling. Es bietet zudem verteiltes GPU-Training, Modell-Speicheroptimierung durch Quantisierung und die Möglichkeit, trainierte Modelle als Inference-Services über API-Endpunkte bereitzustellen.
Features
- Knowledge Graph Construction Tools - Provides a comprehensive framework for building structured knowledge graphs by extracting entities, relations, and events from text.
- Entity and Relation Extraction - Detects and categorizes semantic relationships between entities to transform raw text into structured triples.
- Joint Extractions - Implements a joint extraction process that identifies named entities and their relationships in a single structured output.
- Event Extraction - Identifies event types, trigger words, and argument roles from unstructured text to structure event-based data.
- Knowledge Graph Extraction - Provides a toolkit for extracting entities, relations, and events to build structured knowledge graphs.
- LLM Tool Calling - Utilizes large language models as tool callers through a standardized context protocol to automate data extraction.
- Named Entity Recognition - Implements systems for identifying and classifying named entities using supervised, few-shot, and multimodal models.
- Information Extraction Frameworks - Provides a complete pipeline for converting raw datasets into structured instructions and serving models via API.
- Schema-Driven Extraction - Maps unstructured text to predefined structured formats and task descriptions for domain-specific knowledge extraction.
- LLM-Driven Data Extractors - Leverages large language models as tool callers to automate the transformation of unstructured content into structured formats.
- LLM-to-Structured Data Converters - Employs large language models as tool callers to drive the extraction of specific knowledge into structured formats.
- Custom Model Training - Provides a training pipeline for entity recognition and relation extraction models using customized datasets.
- Distributed GPU Training - Supports spreading neural network training across multiple graphics cards to increase speed and memory capacity.
- Distributed Memory Optimizers - Optimizes model memory through quantization and weight distribution across hardware accelerators to support larger parameter counts.
- Distributed Training - Provides capabilities to train large neural networks across multiple GPUs to handle large datasets and memory requirements.
- Schema-Driven Extraction - Implements a comprehensive system for identifying entities, relations, and events from bilingual text using predefined schemas.
- Relation Extraction Training - Provides a training framework for deep learning models to identify semantic relations between entities.
- Multi-GPU Distribution - Optimizes model memory by splitting weights across multiple graphics cards and using quantization.
- Distant Supervision Pipelines - Leverages existing high-quality triple files to automatically generate relation labels for raw text.
- Tool-Protocol Standardizations - Uses a standardized context protocol to allow language models to act as tool callers for knowledge extraction.
- Training Data Annotations - Offers a system for producing annotated datasets through manual labeling and automated dictionary matching.
- Model Fine-Tuning - Supports fine-tuning deep learning model weights and hyperparameters for domain-specific information extraction.
- Relation Classification - Categorizes semantic relationships between detected entities across document-level and multimodal scenarios.
- Relation Labeling - Provides an automated supervision system for assigning relationship labels to reduce manual training data annotation.
- Instruction-Based Conversions - Transforms raw extraction datasets into structured formats featuring task descriptions and dynamic schemas.
- Model Inference Deployments - Deploys trained extraction models as high-performance API endpoints for real-time inference requests.
- Datenannotation - Generative knowledge graph construction using code models.
- Evaluation and Benchmarking - Instruction-based information extraction dataset for Chinese.
- In Context Learning - Generative knowledge graph construction using code models.
- Named Entity Recognition - Collaborative domain-prefix tuning for cross-domain entity recognition.
- Natural Language Processing - Toolkit for knowledge graph extraction and relation classification.
- Relation Extraction - Unleashing model power for few-shot relation extraction.
- Task-Specific Prompting - Lightweight generative framework for low-resource NER.
- Universal Information Extraction - Code language model for generative knowledge graph construction.
- Zero-Shot Prompting - Generative knowledge graph construction using code models.
