facebookresearch/deitArchived
Deit
DeiT is a PyTorch vision transformer framework designed for image classification. It implements a transformer-based architecture that processes images as sequences of flattened patches using self-attention layers and position-aware sequence modeling instead of convolutional filters.
The project focuses on data-efficient training through a knowledge distillation framework. This system allows a student model to mimic the soft labels of a high-performance teacher model to improve accuracy and generalization, particularly when training on smaller datasets.
The library covers the full development lifecycle, including image classification training, cross-entropy loss optimization, and the deployment of pretrained weights for inference. It also includes a benchmarking tool to evaluate model performance and accuracy against standard datasets.
Features
- Vision Transformers - Implements a vision transformer architecture that processes images as sequences of tokens using self-attention.
- Image Classification Models - Implements a transformer-based deep learning model for categorizing images into predefined classes.
- Classification Training - Implements training workflows for vision transformers to perform image classification using data-efficient methods.
- Patch Embedding Modules - Provides modules to convert images into flattened patch sequences for transformer-based processing.
- Knowledge Distillation - Provides a knowledge distillation workflow to transfer capabilities from a high-performance teacher to a student model.
- Knowledge Distillation Frameworks - Provides a framework for transferring knowledge from teacher models to smaller student networks.
- Teacher-Student Distillation - Implements a teacher-student distillation framework to improve the accuracy of vision transformer models.
- Positional Embedding Layers - Injects learnable positional embeddings into image patch sequences to preserve spatial arrangement.
- PyTorch Vision Transformer Frameworks - Provides a PyTorch-based framework specifically designed for building and training vision transformers.
- Data-Efficient Architectures - Implements a vision transformer optimized for training on smaller datasets using distillation.
- Data-Efficient Transformers - Implements a vision transformer optimized for high accuracy on smaller datasets through distillation.
- Computer Vision Benchmarks - Includes tools for evaluating model accuracy against standard computer vision benchmarking datasets.
- Computer Vision Evaluation Tools - Ships utilities for benchmarking and analyzing the performance of image classification models.
- Bottleneck Layers - Uses bottleneck layers within multi-layer perceptrons to project features after attention blocks.
- Pretrained Model Deployment - Enables the loading and execution of pretrained vision model weights for production inference.
- Classification Accuracy Scorers - Provides utilities to measure classification accuracy and predictive performance during model evaluation.
- Advanced Learning - Data-efficient image transformers and distillation.
- Efficient Vision Architectures - Trains data-efficient image transformers using distillation techniques.
- Efficient Vision Transformers - Distills knowledge into smaller, data-efficient image transformer models.
- Vision Backbones and Classification - Data-efficient image transformers and distillation techniques.
- Vision Transformers - Implements data-efficient image transformers.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
