1 रिपॉजिटरी
Awesome GitHub RepositoriesDocument Preprocessing Pipelines
Transformations that convert raw input data into structured document formats for analysis or chunking.
Distinct from Raw Document Retrieval: The candidates focus on retrieval (fetching) or rendering, not the structural transformation of raw data into a format suitable for chunking.
Explore 1 awesome GitHub repository matching data & databases · Document Preprocessing Pipelines. Refine with filters or upvote what's useful.
Awesome Document Preprocessing Pipelines GitHub Repositories
AI के साथ बेहतरीन रिपॉजिटरी खोजें।हम AI का उपयोग करके सबसे सटीक रिपॉजिटरी खोजेंगे।
- chonkie-inc/chonkie
chonkie-inc/chonkie
4,170GitHub पर देखेंChonkie is a text chunking library designed for retrieval-augmented generation pipelines. It functions as a semantic text splitter and RAG ingestion pipeline, transforming raw text into embedded segments for storage in vector databases. The project distinguishes itself through specialized splitting strategies, including an AST-based code splitter for preserving logical boundaries in source code and a semantic text splitter that uses embedding models to determine boundaries based on meaning. It also provides a vector database ingestor to automate the generation of embeddings and their export t
Transforms raw input into structured document formats to prepare data for the chunking stage.
Pythonaichonkiechunker
