meta-pytorch/captum
Captum
Captum is a model interpretability library for PyTorch that explains why neural networks make the predictions they do. It provides a unified set of attribution and explanation algorithms covering input features, internal layers, individual neurons, and human-understandable concepts, and it also serves as a research framework for building and comparing new interpretability methods.
Feature attribution is the central capability: gradient-based and perturbation-based methods such as Integrated Gradients and DeepLift score how strongly each input feature pushed a prediction up or down, often relative to a baseline input. The same approach extends to layers and neurons to show which parts of a network matter most. Concept-based explanation uses concept vectors to test how higher-level concepts contribute to predictions. A separate training data influence area uses influence functions and similarity comparisons to identify which training examples most shaped a model's behavior on test inputs.
Around these core methods, the library reports a convergence delta so users can judge attribution approximation error, and it can average results over noise-perturbed inputs to produce cleaner attribution maps. Evaluation metrics for infidelity and sensitivity allow different algorithms to be compared on shared models. Visualization tools render attribution scores as plots, and counterfactual and adversarial perturbation tools show how small input edits change a model's output.
