Awesome GitHub RepositoriesEncoder-Decoder Architectures

Donut is an OCR-free document transformer and end-to-end document parser. It functions as a neural network that converts unstructured document images directly into structured data or text without the use of an external optical character recognition engine. The project includes a synthetic document generator to create artificial images and ground-truth labels for training. It employs a transformer model to perform visual question answering and document image classification based on visual layout and text. The system covers several document understanding capabilities, including structured info

BLIP is a vision-language model framework that combines contrastive, matching, and language modeling objectives to align images with text. Built on a multimodal encoder-decoder architecture, it supports distributed data-parallel training with cosine learning rate scheduling and sliding-window metric tracking for training stability. The framework provides capabilities for image captioning, visual question answering, and cross-modal retrieval, scoring semantic alignment between images and text through learned embeddings. It includes toolkits for fine-tuning pre-trained models on custom datasets

Sapiens ist ein hochauflösendes menschliches Sichtmodell, das für hochpräzise, menschenzentrierte Computer-Vision-Aufgaben entwickelt wurde. Es fungiert als Tool-Suite zur Schätzung menschlicher Posen, Tiefe und Oberflächengeometrie. Das Projekt nutzt ein Vision-Transformer-Backbone, um mehrere Aufgaben über einen gemeinsamen Encoder auszuführen. Diese Architektur ermöglicht die gleichzeitige Vorhersage von Skelettstrukturen, Gelenkpositionen und der Entfernung zwischen einer Kamera und einer menschlichen Person. Die Funktionen des Modells decken die Segmentierung menschlicher Körperteile zur Isolierung anatomischer Regionen vom Hintergrund sowie die Vorhersage von Oberflächennormalen zur Wiederherstellung von 3D-Geometriedetails aus 2D-Bildern ab. Diese Aufgaben werden durch ein Multi-Task-Learning-Framework unterstützt, das pixelweise Regression und semantische Segmentierungsmaskierung verwendet.

This is an image segmentation framework and masking toolkit for constructing binary and multi-class neural network architectures. It serves as a deep learning encoder wrapper that integrates pre-trained convolutional neural network architectures into semantic segmentation models. The library enables the use of pre-trained backbones to isolate complex patterns and leverages transfer learning to accelerate training. It provides a collection of overlap-based loss functions and precision metrics specifically designed to evaluate and refine the accuracy of image masks. The toolkit covers the full

Roboflow Sports is a sports video analysis system that combines object detection and tracking with bird's-eye field visualization. Its core pipeline detects and tracks players, referees, and balls across video frames, then maps those tracked positions onto a radar-style overhead view of the playing field. The system goes beyond basic detection by localizing field boundaries and key landmarks such as pitch lines and corners, enabling spatial mapping of player positions relative to the field geometry. It classifies detected players by team affiliation through visual feature extraction and clust

This project is a research framework and toolkit designed for training large-scale vision transformers and multimodal language models. It provides a comprehensive suite for vision-language pretraining, enabling the development of models that map images and text into shared latent spaces. The framework is distinguished by its capabilities in high-fidelity image generation and multimodal research, utilizing normalizing flows and variational autoencoders to produce images from text prompts or class labels. It supports the development of both generative and contrastive models, allowing for a wide

SegFormer is a semantic segmentation framework and transformer-based model designed for pixel-level image classification. It provides a deep learning architecture that assigns class labels to pixels using a hierarchical transformer encoder and a multi-layer perceptron decoder. The framework utilizes a hierarchical transformer encoder to process multi-scale features through a pyramid of blocks and an all-MLP decoder to aggregate these features without complex attention mechanisms. It incorporates overlap patch embedding to preserve local continuity and sequential self-attention reduction to ma

manga-ocr is a Japanese OCR engine and text extraction tool designed to recognize vertical and horizontal Japanese text from manga images. It operates as a vision encoder-decoder model that converts visual text into digital characters. The project includes an OCR training pipeline and a synthetic data generator. These tools create artificial image-text pairs by overlaying diverse Japanese text fonts onto background images to refine recognition models. The system provides automation for extracting text by monitoring the system clipboard or directories. This allows for the conversion of manga