Machine Learning Model Interpretability Tools

SHAP is an explainable AI toolkit that provides a game theoretic framework for interpreting machine learning model predictions. It functions as a feature attribution engine, decomposing model outputs into the sum of individual feature effects to clarify how specific input variables influence a final decision. By assigning importance values to these inputs, the library enables users to understand the logic behind complex predictive models. The project distinguishes itself through its versatility and specialized calculation methods. It operates as a model-agnostic diagnostic library, capable of

SHAP is a machine learning explainer that uses a game-theoretic framework to estimate the contribution of each feature to a model prediction. It provides a set of tools for quantifying how individual input features push a specific output away from a baseline value. The project includes specialized explainers for different architectures, including high-speed implementations for decision trees and ensemble models, linearization algorithms for deep learning networks, and covariance integration for linear models. It also features a model-agnostic interpretability tool that uses a kernel method to

This project is an agnostic model interpretability framework and explainability tool designed to provide local interpretable explanations for individual predictions. It functions as a local surrogate model that approximates the behavior of any machine learning classifier or regression model to identify the most influential features for a specific instance. The framework is designed to be model-agnostic, meaning it can explain predictions across tabular, text, and image data regardless of the underlying architecture. It employs local linear approximations and feature importance visualization t

Interpret is an interpretable machine learning library and glassbox model framework. It provides toolkits for training inherently transparent models and applying post-hoc explanation techniques to make machine learning predictions human-understandable. The framework distinguishes itself by integrating differential privacy into the training of interpretable models to prevent sensitive data from leaking through explanations. It also features a visualization tool for rendering interactive decision paths and model behavior. The library covers model explainability through feature importance calcu

Captum is an open-source library for explaining model predictions by attributing them to input features, neurons, and layers using gradient-based and perturbation-based methods. It provides a modular framework for implementing, evaluating, and combining a range of explanation techniques, including gradient-based attribution, perturbation-based analysis, game-theoretic Shapley value approximation, and surrogate model explanations, with support for parallelization and noise stabilization. The library distinguishes itself through its breadth of attribution methods and its support for advanced in

🔅 Shapash: User-friendly Explainability and Interpretability to Develop Reliable and Transparent Machine Learning Models

Lit is a machine learning interpretability framework and model debugging tool designed to analyze model behavior and performance. It serves as an interpretability dashboard for large language models and a general performance analyzer for text, image, and tabular datasets. The project distinguishes itself through a comprehensive suite of interpretability tools, including salience map generation for feature attribution, the creation of synthetic and counterfactual examples to test robustness, and the projection of high-dimensional embeddings into visual spaces via UMAP or PCA. It further enable

TransformerLens is a library for mechanistic interpretability research designed to reverse engineer the learned algorithms within large language models. It provides a standardized framework for wrapping diverse transformer architectures, allowing researchers to extract, manipulate, and analyze internal activations and weights through a consistent interface. The project distinguishes itself through a comprehensive system of activation hooks that can capture, patch, and ablate internal tensors during the forward pass. It includes specialized utilities for decomposing fused projections, material

A toolbox to iNNvestigate neural networks' predictions!

Code for "High-Precision Model-Agnostic Explanations" paper

Transformers-interpret is a diagnostic library designed for the interpretability of transformer-based machine learning models. It functions as an attribution framework that quantifies the contribution of individual input tokens to a model's final predictions, allowing users to audit decision patterns and debug natural language processing tasks. The library utilizes gradient-based analysis and hook-based introspection to trace how specific input features influence model outputs. By mapping abstract numerical attribution scores back to human-readable linguistic units, it provides a clear view o

This toolkit serves as a framework for interpreting the decision-making processes of graph neural networks. It functions as a library for analyzing how these models process complex network data, providing methods to identify the specific node attributes and structural patterns that influence predictive outcomes. The project distinguishes itself by employing mask-optimized subgraph extraction and gradient-based attribution mapping to isolate the minimal components of a graph that preserve a model's original prediction. By separating graph processing layers from explanation logic, the architect

Vit-explain is a diagnostic framework designed to interpret the decision-making processes of vision transformer models. It functions as a toolkit for inspecting internal model states, allowing users to map visual attention and analyze how specific image features influence classification outcomes. The project distinguishes itself by providing post-hoc model interpretation, which enables the analysis of trained neural networks without requiring architectural modifications or retraining. It employs techniques such as hook-based feature extraction to intercept internal activations during the forw