PAIR-code/lit

Features

• Machine Learning Debugging - Provides interactive visualizations to analyze model predictions and errors and identify failure modes.
• Model Behavioral Analysis - Explores model predictions across text, image, and tabular data via an interactive interface to identify behavioral patterns.
• Model Interpretability - Implements techniques for explaining model decision logic through feature attribution and salience mapping.
• Classification Metrics - Computes standard accuracy and error rates for classification tasks to evaluate model performance.
• Comparative Metric Calculators - Calculates default or custom metrics across the whole dataset or faceted subsets for behavioral comparison.
• Custom Model Integrations - Allows wrapping machine learning models as functions to return predictions and metadata within the visual environment.
• Dataset Integration - Provides interfaces for connecting custom datasets to the interpretability pipeline for slicing and analysis.
• Embedding Visualizations - Projects high-dimensional vector representations into a visual space to identify clusters and relationships in data.
• Synthetic Robustness Example Generation - Creates new input examples using back-translation or adversarial attacks to evaluate model robustness.
• LLM Interpretability Dashboards - Provides a comprehensive visual dashboard for analyzing LLM behavior via salience maps and embeddings.
• Model Comparison Interfaces - Provides interfaces for the side-by-side visual and analytical comparison of outputs from different machine learning models.
• Counterfactual Example Generation - Generates counterfactual and synthetic examples to evaluate model stability and robustness under input perturbations.
• Model Evaluation Metrics - Computes performance statistics across datasets to quantify the accuracy and behavior of trained models.
• ML Model Debugging Tools - Provides a system for inspecting predictions and calculating metrics across text, image, and tabular datasets.
• Model Evaluation Tools - Assesses model performance using a combination of custom metrics, data slicing, and embedding visualizations.
• Model Interpretability Frameworks - Implements a comprehensive toolkit for understanding complex model decisions via feature attribution and decision boundary exploration.
• Model Performance Analysis - Evaluates model predictions and calculates metrics like accuracy or BLEU scores across specific data slices.
• Model Performance Evaluators - Computes accuracy and reliability metrics by comparing model predictions against ground truth labels using confusion matrices.
• Model Prediction Evaluation - Compares model outputs against ground truth labels using field specifications to generate performance reports.
• Embedding Visualizations - Visualizes high-dimensional embeddings in low-dimensional space using techniques like UMAP and PCA.
• Saliency Mapping - Highlights the most influential parts of an input using interpretability methods like Integrated Gradients and LIME.
• Counterfactual Analyses - Observes prediction changes by applying manual edits or automatic transformations to inputs.
• Counterfactual Data Generation - Generates new datapoints by applying transformations like word replacement, scrambling, or back-translation.
• Counterfactual Example Generation - Produces transformed input examples using plugins for word scrambling, regex substitution, and adversarial methods.
• Subset Metric Calculation - Computes performance measures, such as accuracy or BLEU score, across the dataset, current selections, or slices.
• Dimensionality Projection Plots - Visualizes the latent space using UMAP or PCA to identify clusters and patterns in high-dimensional vectors.
• Dataset Filtering and Search - Provides advanced filtering and search using regex, text, and numerical ranges to isolate data subsets.
• Dataset Loading - Implements processes for importing lists of examples and structured data for model evaluation.
• Large Dataset Explorers - Navigates large datasets using a matrix view grouped by feature values with support for zooming and panning.
• Interpretability Pipelines - Uses a modular system of interpreters and generators to process model outputs into salience maps and counterfactuals.
• Side-by-Side Inference Recipes - Implements a side-by-side evaluation system to compare predictions and metrics across the same dataset for multiple models.
• ML Tabular Visualizers - Renders predictions over categorical and numeric tabular data with support for feature attribution.
• Data Selection - Allows users to select datapoints via interactive plots and tables for downstream analysis.
• Concept Importance Quantification - Measures how much a high-level concept contributes to a prediction using Concept Activation Vectors.
• Confusion Matrix Analysis - Provides interactive confusion matrices to compare predictions against gold labels for intersectional analysis.
• Datapoint Editing - Enables modification of input features for selected datapoints to test model sensitivity.
• Decision Boundary Visualizations - Visualizes scalar features and per-class probabilities via scatterplots to identify examples near decision boundaries.
• Decision Threshold Calibration - Optimizes probability thresholds to maximize specific classification metrics like F1 score, accounting for cost and fairness.
• Feature Importance Attribution - Quantifies the influence of specific numeric or categorical features on predictions using partial-dependence plots.
• Reference Datapoint Comparisons - Allows pinning a reference datapoint to visualize how model behavior differs relative to a primary selection.
• Remote Model Integration - Provides the ability to interact with models hosted via network endpoints to maintain computation on the server.
• Evaluation Threshold Gates - Adjusts binary classifier thresholds for specific data slices to optimize for fairness and performance.
• Model Explanation Visualizations - Provides rich visual explanations and salience maps to diagnose the reasoning behind specific model predictions.
• Model Output Visualizers - Renders scatter plots of model scores and regression errors to analyze prediction distributions.
• Prediction Visualization - Renders interactive visualizations for token-aligned output, part-of-speech tags, and dependency parsing edges.
• Classification Error Analysis - Provides a detailed categorical breakdown of classification mistakes using a 2D matrix to identify where models disagree.
• Prompt Salience Analysis - Explains the impact of individual prompt tokens on model outputs using various salience methods.
• Regression Scoring Evaluation - Tracks numerical target scores using regression metrics, bucketed faceting, and scalar scatterplots.
• Pixel Saliency Maps - Generates saliency maps for image inputs using integrated gradients or XRAI to highlight influential pixels.
• Debugging - Visualizes generated text candidates and highlights diffs against reference texts using output probabilities.
• Tabular Counterfactual Generation - Finds alternative data points in tabular datasets to understand model decision boundaries.
• Feature-Based Segmentation - Enables the creation of data slices by faceting along features to analyze performance on subgroups.
• Slice Persistence - Allows users to name and save specific datapoint subsets as reusable slices for quick navigation.
• Tabular Model Explanation - Identifies the importance of specific data columns in tabular datasets using SHAP interpreters.
• Notebook Integrations - Integrates interactive analysis and debugging tools directly within Jupyter notebook environments.
• Notebook Rendering Utilities - Renders the interpretability interface directly within notebook output cells for seamless workflow integration.
• Notebook Tooling - Embeds the interactive interpretability interface directly into notebook cells for inline debugging workflows.
• Interpretability Components - Provides standalone classes for metrics and counterfactual generators to perform analysis independently of a visual interface.
• Client-Server Architecture - Maintains a frontend for interactive visualization and a backend server for heavy model computation and data processing.
• Shared State Management - Encodes the current view and loaded assets into a URL for easy sharing with other users.
• Cross-Component Selection Synchronization - Synchronizes datapoint highlighting across all interactive modules to ensure consistent analysis.
• Partial Dependence Plots - Displays the effect of changing individual features on model output through interactive partial dependence plots.
• Cross-Component Selection Sync - Coordinates data highlighting across disparate visual components by broadcasting selection events via a central state coordinator.
• URL State Synchronization - Serializes the current view configuration and active data filters into the URL to enable shareable application states.
• Profiling View State Encoders - Encodes the current model setup and UI state into a URL for sharing with other users.
• Explainable AI Libraries - Interactive tool for understanding and debugging NLP models.

Star history