nashsu/llm_wiki
Llm Wiki
This project is an LLM knowledge base builder and personal knowledge management tool. It is a desktop application designed to transform diverse documents into a persistent, interlinked wiki through LLM analysis and incremental ingestion.
The system distinguishes itself with a knowledge graph visualizer that uses community detection algorithms to map relationships between concepts and identify topical clusters. It features a hybrid retrieval system that combines keyword matching, vector embeddings, and graph relevance to locate information.
The platform covers a wide range of capabilities including multimodal document parsing for PDF, DOCX, and web content, automated deep research for synthesizing new wiki pages, and a conversational query interface with cited references. It also includes structural analysis tools for knowledge gap analysis and wiki health linting.
The knowledge base is maintained in a directory format compatible with Obsidian vaults.
Features
- LLM Knowledge Base Generators - Transforms diverse documents into a persistent interlinked wiki using LLM analysis and incremental ingestion.
- Cited Query Responses - Provides a conversational interface that retrieves answers from processed knowledge with integrated inline source citations.
- Analysis Ingest Pipelines - Utilizes a chain-of-thought analysis process to identify entities and contradictions during the knowledge base ingestion phase.
- Hybrid Knowledge Retrievers - Implements a hybrid retrieval system combining keyword matching, vector embeddings, and graph relevance to locate information.
- Knowledge Ingestion Chains - Processes documents through separate extraction and generation phases to create structured wiki pages with strict source traceability.
- Personal Knowledge Bases - Provides a system for organizing documents and web clips into a structured wiki with automatic linking and source traceability.
- Automated Wiki Generations - Uses a multi-step LLM chain to build a persistent, interlinked wiki from documents while maintaining source traceability.
- Graph Community Detection - Uses modularity-based community detection algorithms to automatically discover and group related knowledge clusters.
- Document Parsing Engines - Provides a high-fidelity parser that converts PDF, DOCX, PPTX, and XLSX files into structured Markdown.
- Hybrid Vector-Graph Databases - Combines vector embeddings and knowledge graph relationships to provide high-precision hybrid information retrieval.
- Multi-Format Document Ingestion - Supports the ingestion and normalization of PDF, DOCX, and Markdown formats for a unified knowledge base.
- Hybrid Retrieval - Implements a search pipeline combining keyword matching, vector semantic embeddings, and graph-based expansion.
- Semantic Search - Provides an embedding-based retrieval system for finding semantically related pages via a compatible endpoint.
- Knowledge Graph Visualizations - Provides an interactive interface for mapping relationships between concepts and visualizing the network topology of the knowledge base.
- Cited Conversation Histories - Offers independent chat sessions with cited references to wiki pages and the ability to archive synthesized answers.
- Deep Research Execution - Generates optimized web search queries and synthesizes findings into new wiki pages using autonomous analytical agents.
- Human-in-the-Loop Systems - Integrates human oversight into the ingestion pipeline to resolve ambiguous items flagged by the LLM.
- Knowledge Gap Analysis - Provides a structural analysis tool that identifies isolated pages and surprising connections to surface research needs.
- Model Context Protocol Servers - Implements a local server based on the Model Context Protocol to expose internal data and tools to compatible AI clients.
- Knowledge Base Visualizers - Creates a visual map of the knowledge base based on direct links, source overlap, and type affinity.
- Community Detection Algorithms - Employs a clustering algorithm to group related pages into communities to discover natural knowledge domains.
- Web Article Extraction - Includes a utility to extract clean article text and core content from web pages for automatic integration into the knowledge base.
- Web Clipping Extractors - Ships a browser extension for capturing specific web page segments and stripping clutter to trigger the ingestion pipeline.
- Automated Document Ingestion - Implements a crash-resilient sequential queue for importing and transforming diverse file formats into the knowledge base.
- Multimodal PDF Indexers - Provides a system for extracting embedded images from PDFs and using vision models to generate searchable factual captions.
- File System Watchers - Provides a background service that monitors local folders to automate knowledge base updates based on file changes.
- Note Relationship Mappings - Provides a graphical representation of wiki links and entity connections to visualize knowledge relationships.
- Filesystem Event Synchronization - Monitors local directories for filesystem changes to automatically trigger knowledge base ingestion or cleanup.
- AI Output Review Queues - Includes an asynchronous queue for users to validate and modify model-generated wiki items before they are persisted.
