# nvlabs/segformer

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/nvlabs-segformer).**

3,347 stars · 423 forks · Python · other

## Links

- GitHub: https://github.com/NVlabs/SegFormer
- Homepage: https://arxiv.org/abs/2105.15203
- awesome-repositories: https://awesome-repositories.com/repository/nvlabs-segformer.md

## Topics

`ade20k` `cityscapes` `semantic-segmentation` `transformer`

## Description

SegFormer is a semantic segmentation framework and transformer-based model designed for pixel-level image classification. It provides a deep learning architecture that assigns class labels to pixels using a hierarchical transformer encoder and a multi-layer perceptron decoder.

The framework utilizes a hierarchical transformer encoder to process multi-scale features through a pyramid of blocks and an all-MLP decoder to aggregate these features without complex attention mechanisms. It incorporates overlap patch embedding to preserve local continuity and sequential self-attention reduction to manage computational costs.

The project covers the full lifecycle of computer vision model development, including GPU-accelerated training, model evaluation using pixel-level accuracy metrics, and inference for generating color-coded visualization maps. It includes utilities for loading pretrained weights and checkpoints to facilitate scene understanding and object masking.

A Dockerized runtime environment is provided to ensure reproducible deployment and consistent execution of the model and its dependencies.

## Tags

### Artificial Intelligence & ML

- [Hierarchical Encoders](https://awesome-repositories.com/f/artificial-intelligence-ml/transformer-encoders/hierarchical-encoders.md) — Utilizes a hierarchical transformer encoder to process multi-scale features through a pyramid of blocks.
- [Image Segmentation](https://awesome-repositories.com/f/artificial-intelligence-ml/computer-vision-systems/image-segmentation.md) — Runs pretrained models on images to produce pixel-level class predictions and color-coded visualization maps.
- [Computer Vision Training](https://awesome-repositories.com/f/artificial-intelligence-ml/computer-vision-training.md) — Implements standardized training routines to optimize network parameters for pixel-level categorization within image datasets.
- [Pretrained Model Snapshots](https://awesome-repositories.com/f/artificial-intelligence-ml/custom-model-training/pretrained-model-integrations/pretrained-model-snapshots.md) — Provides pretrained weights and checkpoints for performing scene understanding and object masking without training from scratch.
- [GPU-Accelerated Training](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/machine-learning-training/gpu-accelerated-training.md) — Provides a pipeline for optimizing model weights using single or multiple GPUs to handle large-scale transformer computations.
- [PyTorch Semantic Segmentation Libraries](https://awesome-repositories.com/f/artificial-intelligence-ml/pytorch-semantic-segmentation-libraries.md) — Provides a complete framework for training, evaluating, and visualizing pixel-level image classification using PyTorch.
- [Segmentation Metrics](https://awesome-repositories.com/f/artificial-intelligence-ml/segmentation-metrics.md) — Computes standard pixel-level segmentation metrics to measure prediction accuracy on test datasets. ([source](https://cdn.jsdelivr.net/gh/nvlabs/segformer@master/README.md))
- [Semantic Segmentation Architectures](https://awesome-repositories.com/f/artificial-intelligence-ml/vision-transformers/encoder-decoder-architectures/semantic-segmentation-architectures.md) — Implements a deep learning architecture that assigns class labels to pixels using a hierarchical transformer encoder and MLP decoder.
- [Containerized ML Environments](https://awesome-repositories.com/f/artificial-intelligence-ml/containerized-ml-environments.md) — Ships a containerized setup to ensure reproducible deployment and inference of transformer-based computer vision models.
- [Decoder Architectures](https://awesome-repositories.com/f/artificial-intelligence-ml/decoder-architectures.md) — Ships a lightweight all-MLP decoder that aggregates multi-level features without the need for complex attention mechanisms.
- [Overlap Patch Embeddings](https://awesome-repositories.com/f/artificial-intelligence-ml/image-convolution-operations/image-patch-embedders/patch-embedding-modules/overlap-patch-embeddings.md) — Incorporates overlap patch embedding to preserve local continuity and reduce boundary artifacts during tokenization.
- [Multi-Scale Feature Aggregation](https://awesome-repositories.com/f/artificial-intelligence-ml/inference-scaling/resolution-scaling/hierarchical-feature-pyramids/multi-scale-feature-pyramids/multi-scale-feature-aggregation.md) — Combines features from multiple encoder stages to capture both fine details and coarse semantics for pixel-level classification.
- [Vision Model Weight Loading](https://awesome-repositories.com/f/artificial-intelligence-ml/large-scale-model-training/vision-transformer-pre-training/pre-trained-model-checkpoints/vision-model-weight-loading.md) — Implements utilities for loading pretrained weights and checkpoints specifically for computer vision architectures to initialize networks. ([source](https://github.com/NVlabs/SegFormer/tree/master/resources))
- [MLP Decoders](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/frameworks/model-construction/neural-network-layers/convolution-layers/layered-architectures/multi-layer-perceptrons/mlp-decoders.md) — Implements an all-MLP decoder to aggregate multi-level features for generating segmentation masks without complex attention.
- [Segmentation Visualizations](https://awesome-repositories.com/f/artificial-intelligence-ml/segmentation-visualizations.md) — Provides tools to overlay predicted pixel-level class labels as color-coded masks on original images for visual verification. ([source](https://github.com/NVlabs/SegFormer/blob/master/README.md))

### Testing & Quality Assurance

- [Model Evaluation](https://awesome-repositories.com/f/testing-quality-assurance/model-testing/model-evaluation.md) — Measures prediction accuracy and performance by testing trained models against benchmark datasets using GPU acceleration.

### Part of an Awesome List

- [Sequential Self-Attention Reduction](https://awesome-repositories.com/f/awesome-lists/ai/attention-mechanisms/self-attention-implementations/sequential-self-attention-reduction.md) — Reduces computational cost by applying self-attention sequentially across spatial dimensions rather than jointly.
- [Object Detection and Segmentation](https://awesome-repositories.com/f/awesome-lists/ai/object-detection-and-segmentation.md) — Simple and efficient design for semantic segmentation.

### DevOps & Infrastructure

- [Docker Container Deployments](https://awesome-repositories.com/f/devops-infrastructure/container-orchestration/container-runtimes/runtime-configuration-interfaces/docker-socket-orchestrators/docker-target-configurators/docker-container-deployments.md) — Provides a Dockerized runtime environment to package the model and dependencies for consistent execution and reproducible deployment. ([source](https://github.com/NVlabs/SegFormer/tree/master/docker))
- [Containerized Deployments](https://awesome-repositories.com/f/devops-infrastructure/containerized-deployments.md) — Packages the computer vision environment into a Docker image to ensure reproducible inference and consistent execution.