How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Fast Multimodal Semantic Deduplication & Filtering
The main features of minishlab/semhash are: Data Curation and Filtering, LLM Development Tools, Training Datasets, Data Processing, Data Processing Tools.
Projects with overlapping indexed features include: huggingface/datatrove — Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline… jpmens/jo — Jo is a command-line utility designed to construct and manipulate JSON objects and arrays directly from shell… catchthetornado/pdf-extract-api — Document (PDF, Word, PPTX ...) extraction and parse API using state of the art modern OCRs + Ollama supported models.… allenai/olmocr — Olmocr is a distributed document processing framework designed to convert PDF and image files into structured… bytedance/dolphin — Dolphin is a multimodal layout analyzer and image-to-structure converter that transforms photographed or digital… 599yongyang/datasetloom — 一个面向多模态大模型训练的智能数据集构建与评估平台.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Jo is a command-line utility designed to construct and manipulate JSON objects and arrays directly from shell arguments and standard input. It functions as a data processing tool that transforms raw input into structured formats, enabling the generation of complex payloads for APIs, configuration files, and automated data pipelines. The tool distinguishes itself through its ability to resolve hierarchical data structures using delimiter-based path definitions and its integrated type-inference engine, which automatically casts input values into native boolean, numeric, or null types. Users can
Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.
