snorkel-team/snorkel
Snorkel
Snorkel is a weak supervision system that enables users to programmatically generate training labels for machine learning models without manual annotation. At its core, it provides a framework for writing labeling functions as Python callables that each vote on data points, and then trains a probabilistic graphical model over these multiple weak supervision sources to estimate latent true labels without any ground truth data.
The system automatically learns accuracy and correlation parameters between labeling functions by analyzing observed agreement patterns on unlabeled data, converting labeling function outputs into a sparse label matrix for factor graph inference that resolves conflicts and correlations. It outputs calibrated probabilistic labels per data point by marginalizing over the generative model's posterior distribution, and supports training downstream classifiers using these probabilistically labeled data. Snorkel also includes slice-based model evaluation, which partitions training data into meaningful subgroups to monitor and improve downstream classifier performance on specific cohorts.
Beyond its core labeling capabilities, Snorkel provides tools for label quality diagnosis, allowing users to analyze conflicts, accuracy, and coverage of labeling functions to debug and refine weak supervision sources. It supports transformation-based data augmentation, applying user-defined transformation functions to existing labeled examples to synthetically expand training data diversity. The system also includes preprocessing pipelines for both labeling and model input, ensuring data is in the expected format before labeling functions or model training run.
Features
- Programmatic Weak Supervision Labelers - Applies user-defined labeling functions to unlabeled data to produce probabilistic training labels without manual annotation.
- Weak Supervision - Generates training labels programmatically from multiple noisy sources like heuristics and knowledge bases.
- Weak Supervision Label Models - Trains a probabilistic graphical model over multiple weak supervision sources to estimate latent true labels.
- Labeling Function Diagnostics - Provides metrics to debug and refine labeling functions by analyzing accuracy, coverage, and conflicts.
- Weak Supervision Classifier Training - Trains discriminative classifiers using probabilistic labels generated from labeling functions to combine weak signals into a strong model.
- Label Calibration Models - Outputs calibrated probabilistic labels per data point by marginalizing over the generative model's posterior.
- Weak Supervision Classifier Training - Trains downstream classifiers using probabilistically labeled data from weak supervision sources.
- Source Combiners - Combines multiple weak supervision sources like heuristics and knowledge bases to create training labels programmatically.
- Labeling Function Abstractions - Provides a framework for writing labeling functions as Python callables that vote on data points.
- Weak Supervision Label Diagnostics - Computes summary statistics and diagnostics on generated labels to help debug and refine labeling functions.
- Labeling Function Frameworks - Provides a framework for writing Python callables that vote on data points to programmatically label datasets.
- Label Function Dependency Analyzers - Automatically learns accuracy and correlation parameters between labeling functions from agreement patterns.
- Generative Label Model Trainers - Trains a generative graphical model to estimate latent true labels from multiple noisy weak supervision sources.
- Probabilistic Label Training - Trains downstream classifiers using probabilistically labeled data produced by weak supervision sources.
- Label Matrix Aggregators - Converts labeling function outputs into a sparse label matrix for factor graph inference.
- Training Data Augmentation - Applies transformations to existing labeled examples to increase dataset diversity and improve model robustness.
- Slice-Based Evaluators - Evaluates downstream classifier performance on user-defined data slices to identify systematic failures.
- Slice-Based Model Evaluators - Partitions training data into meaningful subgroups to monitor and improve classifier performance on specific cohorts.
- Subgroup Performance Evaluators - Slices datasets into meaningful subgroups and evaluates model performance on each to uncover systematic failures.
- Weak Supervision Label Diagnostics - Diagnoses label quality and conflicts to refine labeling functions for better accuracy.
- User-Defined Augmentation Transformers - Applies user-defined transformation functions to existing labeled examples to expand training data diversity.
- User-Defined Augmentations - Applies user-defined transformation functions to existing labeled examples to expand training data diversity.
- Model Performance Slicing - Partitions training data into meaningful subgroups to monitor and improve downstream classifier performance on specific cohorts.
- Weak Supervision Label Diagnostics - Analyzes conflicts, accuracy, and coverage of labeling functions to debug weak supervision sources.
- Weak Supervision Label Diagnostics - Analyzes conflicts and quality of automatically generated labels to refine labeling functions.
- Data Centric AI - Programmatically generates training data using weak supervision.
- Text Annotation Tools - System for programmatically building and managing training data.
- Data Enrichment - System for generating training data using weak supervision.
- Data Loading Extraction - Generating training data with weak supervision.
