How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
Fast Multimodal Semantic Deduplication & Filtering
Cleanlab is a data-centric AI library and toolkit designed to improve machine learning model performance by detecting label errors and increasing overall dataset quality. It implements a confident learning framework that iteratively refines label noise estimates by comparing model predictions with estimated label probabilities to identify mislabeled examples. The project provides specialized utilities for active learning optimization, allowing for the selection of the most impactful examples for labeling or re-labeling. It also includes an outlier detection tool to identify atypical data poin
The data scientist's open-source choice to scale, assess and maintain natural language data. Treat training data like a software artifact.
Argilla is a collaborative AI feedback tool and data curation management system. It serves as a human-in-the-loop dataset platform designed to coordinate workforce annotators and domain experts in labeling, rating, and refining data samples for machine learning projects. The platform focuses on large language model dataset curation and reinforcement learning from human feedback workflows. It provides a shared workspace for integrating human expertise into AI development to validate model outputs and correct data errors. The system manages the end-to-end machine learning data pipeline, includ
Scalable data pre processing and curation toolkit for LLMs
The main features of nvidia/nemo-curator are: Data Curation and Filtering, LLM Development Tools, Data Annotation and Synthesis.
Open-source alternatives to nvidia/nemo-curator include: minishlab/semhash — Fast Multimodal Semantic Deduplication & Filtering. cleanlab/cleanlab — Cleanlab is a data-centric AI library and toolkit designed to improve machine learning model performance by detecting… code-kern-ai/refinery — The data scientist's open-source choice to scale, assess and maintain natural language data. Treat training data like… cvat-ai/cvat — CVAT is an open-source, web-based platform designed for annotating images, videos, and 3D point clouds to create… diyago/tabular-data-generation — We well know GANs for success in the realistic image generation. However, they can be applied in tabular data… argilla-io/argilla — Argilla is a collaborative AI feedback tool and data curation management system. It serves as a human-in-the-loop…