chonkie-inc/chonkie

Features

• Text Chunks - Provides a comprehensive library for dividing documents into semantic, structural, or token-based chunks for RAG pipelines.
• Document Chunking & Embedding - Orchestrates the document preparation stage including splitting text into overlapping segments and generating vector embeddings.
• RAG Data Pipelines - Implements workflows for preparing, cleaning, splitting, and embedding unstructured text for retrieval-augmented generation.
• Code-Aware Chunking - Uses abstract syntax tree parsing to split source code into chunks that preserve logical class and function boundaries.
• Contextual Chunking - Enriches text segments with overlapping neighboring chunks or embeddings to improve retrieval accuracy in RAG pipelines.
• Text Embedding Generators - Transforms text into numerical vector representations using pre-trained transformer models.
• Vector Embeddings - Converts text chunks into vector representations using external APIs to enable semantic-based splitting.
• Semantic Vector Search - Implements retrieval of text chunks based on the mathematical distance between query and document embeddings.
• Workflow Orchestration - Coordinates the sequential execution of data fetching, preprocessing, chunking, and exporting within AI pipelines.
• Vector Database Exporters - Generates text embeddings and exports processed document data into vector databases for RAG workflows.
• AST-Based Code Splitting - Divides programming files into chunks using AST parsing to preserve the logical boundaries of classes and functions.
• Document Chunking and Embedding Pipelines - Executes automated pipelines that sequentially split documents, generate embeddings, and store them in vector databases.
• Semantic Chunking Groupers - Divides text into segments based on meaning using embedding models and similarity thresholds.
• Text Processing Pipelines - Provides named, reusable modular workflows for transforming raw text into structured segments.
• Semantic Segmenters - Uses embedding models and neural networks to determine optimal text boundaries based on meaning.
• Vector Database Integrations - Implements a consistent adapter interface to map text chunks and embeddings across multiple third-party vector databases.
• Embedding Ingestion Pipelines - Implements a workflow orchestrator that transforms raw text into embedded chunks for insertion into vector stores.
• Semantic Splitters - Determines optimal text boundaries based on meaning using embedding models and neural networks.
• Data Pipeline Orchestrators - Orchestrates sequential data fetching, preprocessing, chunking, and exporting through reusable pipeline chains.
• Byte-Size Splitting - Segments text into chunks based on byte size limits using SIMD acceleration for high throughput.
• Hierarchical Chunking - Chunks structured documents using a sequence of delimiters like paragraphs, sentences, and words.
• Image Text Extractions - Provides capabilities for recognizing and extracting text from images and PDFs using local and cloud OCR engines.
• LLM API Integrations - Integrates with external large language model APIs to perform content summarization and analysis during chunking.
• Recursive Text Splitting - Breaks long, structured documents into smaller segments by recursively applying splitting rules.
• Agentic Splitting - Uses generative models to analyze text structure and identify optimal split points based on semantic understanding.
• API-Based Splitting - Segments text using specialized external APIs for domain-specific requirements such as legal documents.
• Chunk Refinement - Implements post-processing enhancements to text segments to improve overall retrieval quality.
• Tabular Data Chunking - Provides specialized chunking for markdown and HTML tables that preserves headers across segments.
• Similarity Signal Smoothing - Applies Savitzky-Golay filtering to embedding similarity data to stabilize semantic boundaries and reduce noise.
• BERT-Based Splitting - Uses a fine-tuned BERT model to detect semantic shifts and determine topic-coherent boundaries.
• Late Chunking - Generates document-level embeddings before deriving individual chunk representations to maintain global context.
• Document-Level Contextual Chunking - Splits text into segments using document-level embeddings to retain broad context from the original text.
• Sentence Boundary Detection - Divides text into chunks while preserving complete sentence boundaries to maintain context.
• Vector Nearest Neighbor Search - Retrieves the most similar text chunks from an Elasticsearch index using k-Nearest Neighbor search.
• MongoDB Exporters - Writes processed text segments and embeddings into MongoDB NoSQL collections.
• Elasticsearch Exporters - Embeds text chunks and stores them in Elasticsearch indices with automated field mapping.
• Milvus Exporters - Embeds text chunks and stores them in Milvus collections with automatic schema creation.
• PostgreSQL Vector Stores - Saves processed text segments and vector embeddings into PostgreSQL using the pgvector extension.
• Embedding Service Integrations - Connects to various embedding models through a consistent interface to maintain provider compatibility.
• Asynchronous Processing - Utilizes non-blocking asynchronous calls to handle high-throughput text ingestion and concurrent API requests.
• Asynchronous Ingestion Pipelines - Processes text workflows using non-blocking asynchronous calls to handle high-throughput batch ingestion.
• Qdrant Integrations - Embeds text segments and stores them directly into Qdrant vector collections.
• Vector Store Orchestrators - Manages the process of embedding text segments and exporting them to various managed vector databases.
• Vector Indexing - Creates and configures database indices optimized for high-dimensional vector data to speed up similarity searches.
• Custom Embedding Logic - Provides a base class and registration system for defining custom embedding handlers.
• CLI Execution - Provides a command-line interface to execute complete document preprocessing and chunking pipelines.
• Batch Document Processing - Handles multiple documents in a single operation to efficiently produce chunks for large collections.

Historique des stars