Awesome GitHub RepositoriesVision Transformers

This project is an educational platform and research toolkit designed to teach deep learning through a combination of mathematical theory, visual diagrams, and executable code. It provides a comprehensive environment for building, training, and evaluating neural networks, grounding complex concepts in interactive computational notebooks that allow for hands-on experimentation. The framework distinguishes itself by interleaving theoretical foundations—including linear algebra, calculus, and probability—with practical implementations across multiple industry-standard libraries. It supports flex

This library provides a comprehensive collection of modular building blocks and research-backed architectures for implementing vision transformers within the PyTorch framework. It serves as a centralized repository for constructing, training, and analyzing attention-based models, offering a wide array of specialized variants designed for image classification and visual representation learning. The project distinguishes itself through a focus on architectural efficiency and flexibility, supporting diverse input formats including non-square images and volumetric data like video. It incorporates

This project is a comprehensive deep learning framework and educational platform designed for constructing, training, and evaluating neural network architectures. It provides a modular environment for building models through tensor operations and automatic differentiation, supporting a wide range of tasks from image classification and object detection to sequential data processing. Beyond its core technical capabilities, the project distinguishes itself by integrating professional career development resources directly into its learning ecosystem. It offers structured guidance, resume reviews,

Qwen3-VL is a multimodal vision-language model designed to process and reason across images, videos, and text. It functions as a computer vision framework capable of identifying objects, extracting structured data from documents, and interpreting spatial elements within visual media. The system operates as an automated user interface interaction agent, interpreting screen data to navigate software and mobile applications. By utilizing a unified transformer architecture, it performs complex visual reasoning to execute user-defined tasks without manual input. Beyond interface navigation, the m

LaTeX-OCR is a specialized optical character recognition system designed to identify and transcribe complex mathematical symbols and their spatial relationships from images. It functions as a machine learning engine that converts visual representations of equations into structured LaTeX code for use in technical documentation and academic typesetting. The project utilizes a hierarchical vision-based encoding and autoregressive sequence decoding architecture to process input images and generate mathematical notation token by token. Beyond its core recognition capabilities, the system provides

Swin-Transformer is a deep learning framework designed for training and deploying hierarchical vision transformer models. It serves as a research library and toolkit for computer vision tasks, providing the infrastructure to build models that replace standard convolution operations with sliding window self-attention mechanisms. By utilizing a multi-scale feature hierarchy, the framework enables the processing of visual data at varying resolutions and spatial scales. The project distinguishes itself through its implementation of shifted window partitioning, which facilitates global information

PaddleDetection is an object detection framework designed for the end-to-end development, training, and deployment of computer vision models. It provides a comprehensive library of modular neural network architectures and pipelines that support object detection, instance segmentation, and multi-object tracking tasks. The project distinguishes itself through a configuration-driven approach that decouples model components like backbones and heads, allowing for the flexible assembly of custom vision workflows. It incorporates advanced techniques such as anchor-free detection logic, joint detecti

This project is a research library and toolkit for deep learning computer vision, focused on implementing transformer and mixer-based architectures for image classification. It processes visual data by converting images into sequences of patches, allowing standard attention mechanisms to capture global dependencies without relying on traditional convolutional operations. The framework distinguishes itself through its support for multimodal embedding analysis, which maps images and text into a shared latent vector space. This capability enables zero-shot classification and cross-modal retrieva

This project is a self-supervised vision foundation model based on a vision transformer architecture. It is designed to learn dense visual representations from unlabeled images, serving as a general-purpose backbone for a wide variety of downstream vision tasks. The system is distinguished by its use of self-distillation and masked image modeling to extract semantic and geometric features. It also incorporates an image-text alignment model that maps visual embeddings to textual descriptions, enabling zero-shot image recognition, zero-shot segmentation, and cross-modal retrieval. The project

This is a PyTorch library and framework for self-supervised vision learning. It provides an implementation of masked autoencoders and vision transformers designed to learn image representations by reconstructing masked image patches from unlabeled data. The project features a distributed training pipeline that scales workloads across multiple GPU nodes. This infrastructure includes multi-node orchestration and gradient accumulation to manage large batch sizes and coordinate resource requests across clusters. The toolkit covers a complete workflow from self-supervised masked pre-training to d

This project is a PyTorch vision transformer framework designed for self-supervised learning. It implements a model that trains visual representations using a momentum teacher and self-distillation without the need for labeled data. The library functions as an image feature extractor and visual attention visualizer, allowing for the generation of high-dimensional vectors and the rendering of self-attention maps as heatmaps or videos to analyze model focus. It provides comprehensive tools for downstream vision evaluation, including linear probe classification, k-nearest neighbor categorizatio

Donut is an OCR-free document transformer and end-to-end document parser. It functions as a neural network that converts unstructured document images directly into structured data or text without the use of an external optical character recognition engine. The project includes a synthetic document generator to create artificial images and ground-truth labels for training. It employs a transformer model to perform visual question answering and document image classification based on visual layout and text. The system covers several document understanding capabilities, including structured info

x-transformers ist eine PyTorch-Bibliothek und ein Research-Toolkit für den Aufbau von Transformer-Architekturen. Es bietet ein modulares Framework für die Implementierung experimenteller Transformer-Forschung, einschließlich einer Suite fortschrittlicher Attention-Mechanismen, Tools für die Modellierung langer Sequenzen und eines Frameworks für Vision-Transformer. Das Projekt zeichnet sich durch den Fokus auf speichereffiziente und performante Komponenten aus, wie etwa Flash-Attention mit Tiled-Kernels und Multi-Query-Attention. Zudem implementiert es spezialisierte Methoden zur Erweiterung von Kontextfenstern, einschließlich Sequence-Recurrence und Rotary-Positional-Embeddings. Die Bibliothek deckt ein breites Spektrum architektonischer Funktionen ab, darunter verschiedene Normalisierungsschemata zur Stabilisierung des Trainings, Gated-Feedforward-Netzwerke und benutzerdefinierte Layer-Topologien wie Macaron-Netzwerke. Sie unterstützt sowohl Encoder- als auch Decoder-Konstruktionen und bietet Tools für die autoregressive Sequenzgenerierung sowie Vision-Language-Aufgaben wie Bildunterschriften.

MobileSAM ist ein leichtgewichtiger Bildsegmentierer und ein prompt-basiertes Vision-Modell, das für die schnelle Objektisolierung auf Hardware mit begrenzten Ressourcen entwickelt wurde. Es fungiert als automatisches Bildmaskierungstool, das in der Lage ist, einzelne Objekte in einem gesamten Bild ohne manuelle Eingabe zu erkennen und zu isolieren. Das System ermöglicht eine prompt-basierte Objektmaskierung unter Verwendung von Koordinatenpunkten oder Begrenzungsrahmen, um präzise Masken zu generieren. Es unterstützt zudem die Segmentierung aller Objekte in einem Bild durch objektbewusstes Prompt-Sampling, um jedes einzelne Objekt in einer Szene zu identifizieren. Um das Deployment auf Mobilgeräten und Edge-Geräten zu erleichtern, ist das Modell mit ONNX-Export kompatibel, wodurch das Vision-Modell auf verschiedenen plattformübergreifenden Hardware-Runtimes ausgeführt werden kann.

BLIP is a vision-language model framework that combines contrastive, matching, and language modeling objectives to align images with text. Built on a multimodal encoder-decoder architecture, it supports distributed data-parallel training with cosine learning rate scheduling and sliding-window metric tracking for training stability. The framework provides capabilities for image captioning, visual question answering, and cross-modal retrieval, scoring semantic alignment between images and text through learned embeddings. It includes toolkits for fine-tuning pre-trained models on custom datasets

Sapiens ist ein hochauflösendes menschliches Sichtmodell, das für hochpräzise, menschenzentrierte Computer-Vision-Aufgaben entwickelt wurde. Es fungiert als Tool-Suite zur Schätzung menschlicher Posen, Tiefe und Oberflächengeometrie. Das Projekt nutzt ein Vision-Transformer-Backbone, um mehrere Aufgaben über einen gemeinsamen Encoder auszuführen. Diese Architektur ermöglicht die gleichzeitige Vorhersage von Skelettstrukturen, Gelenkpositionen und der Entfernung zwischen einer Kamera und einer menschlichen Person. Die Funktionen des Modells decken die Segmentierung menschlicher Körperteile zur Isolierung anatomischer Regionen vom Hintergrund sowie die Vorhersage von Oberflächennormalen zur Wiederherstellung von 3D-Geometriedetails aus 2D-Bildern ab. Diese Aufgaben werden durch ein Multi-Task-Learning-Framework unterstützt, das pixelweise Regression und semantische Segmentierungsmaskierung verwendet.

This is an image segmentation framework and masking toolkit for constructing binary and multi-class neural network architectures. It serves as a deep learning encoder wrapper that integrates pre-trained convolutional neural network architectures into semantic segmentation models. The library enables the use of pre-trained backbones to isolate complex patterns and leverages transfer learning to accelerate training. It provides a collection of overlap-based loss functions and precision metrics specifically designed to evaluate and refine the accuracy of image masks. The toolkit covers the full

Roboflow Sports is a sports video analysis system that combines object detection and tracking with bird's-eye field visualization. Its core pipeline detects and tracks players, referees, and balls across video frames, then maps those tracked positions onto a radar-style overhead view of the playing field. The system goes beyond basic detection by localizing field boundaries and key landmarks such as pitch lines and corners, enabling spatial mapping of player positions relative to the field geometry. It classifies detected players by team affiliation through visual feature extraction and clust

DeiT ist ein PyTorch-Vision-Transformer-Framework, das für die Bildklassifizierung entwickelt wurde. Es implementiert eine Transformer-basierte Architektur, die Bilder als Sequenzen abgeflachter Patches unter Verwendung von Self-Attention-Layern und positionsbewusster Sequenzmodellierung anstelle von konvolutiven Filtern verarbeitet. Das Projekt konzentriert sich auf dateneffizientes Training durch ein Knowledge-Distillation-Framework. Dieses System ermöglicht es einem Studentenmodell, die Soft-Labels eines leistungsstarken Lehrermodells nachzuahmen, um Genauigkeit und Generalisierung zu verbessern, insbesondere beim Training mit kleineren Datensätzen. Die Bibliothek deckt den gesamten Entwicklungslebenszyklus ab, einschließlich Training zur Bildklassifizierung, Optimierung der Cross-Entropy-Loss-Funktion und Bereitstellung vortrainierter Gewichte für die Inferenz. Sie enthält zudem ein Benchmarking-Tool zur Bewertung der Modellleistung und -genauigkeit anhand von Standarddatensätzen.

This project is a research framework and toolkit designed for training large-scale vision transformers and multimodal language models. It provides a comprehensive suite for vision-language pretraining, enabling the development of models that map images and text into shared latent spaces. The framework is distinguished by its capabilities in high-fidelity image generation and multimodal research, utilizing normalizing flows and variational autoencoders to produce images from text prompts or class labels. It supports the development of both generative and contrastive models, allowing for a wide