Awesome GitHub RepositoriesEncoder-Decoder Architectures

Donut is an OCR-free document transformer and end-to-end document parser. It functions as a neural network that converts unstructured document images directly into structured data or text without the use of an external optical character recognition engine. The project includes a synthetic document generator to create artificial images and ground-truth labels for training. It employs a transformer model to perform visual question answering and document image classification based on visual layout and text. The system covers several document understanding capabilities, including structured info

BLIP is a vision-language model framework that combines contrastive, matching, and language modeling objectives to align images with text. Built on a multimodal encoder-decoder architecture, it supports distributed data-parallel training with cosine learning rate scheduling and sliding-window metric tracking for training stability. The framework provides capabilities for image captioning, visual question answering, and cross-modal retrieval, scoring semantic alignment between images and text through learned embeddings. It includes toolkits for fine-tuning pre-trained models on custom datasets

Sapiens est un modèle de vision humaine haute résolution conçu pour des tâches de vision par ordinateur centrées sur l'humain de haute précision. Il fonctionne comme une suite d'outils pour estimer la pose humaine, la profondeur et la géométrie de surface. Le projet utilise une architecture vision transformer comme backbone pour effectuer plusieurs tâches via un encodeur partagé. Cette architecture permet la prédiction simultanée des structures squelettiques, des emplacements des articulations et de la distance entre une caméra et un sujet humain. Les capacités du modèle couvrent la segmentation des parties du corps humain pour isoler les régions anatomiques des arrière-plans et la prédiction des normales de surface pour récupérer les détails géométriques 3D à partir d'images 2D. Ces tâches sont soutenues par un framework d'apprentissage multi-tâches qui utilise la régression au niveau des pixels et le masquage par segmentation sémantique.

This is an image segmentation framework and masking toolkit for constructing binary and multi-class neural network architectures. It serves as a deep learning encoder wrapper that integrates pre-trained convolutional neural network architectures into semantic segmentation models. The library enables the use of pre-trained backbones to isolate complex patterns and leverages transfer learning to accelerate training. It provides a collection of overlap-based loss functions and precision metrics specifically designed to evaluate and refine the accuracy of image masks. The toolkit covers the full

Roboflow Sports is a sports video analysis system that combines object detection and tracking with bird's-eye field visualization. Its core pipeline detects and tracks players, referees, and balls across video frames, then maps those tracked positions onto a radar-style overhead view of the playing field. The system goes beyond basic detection by localizing field boundaries and key landmarks such as pitch lines and corners, enabling spatial mapping of player positions relative to the field geometry. It classifies detected players by team affiliation through visual feature extraction and clust

This project is a research framework and toolkit designed for training large-scale vision transformers and multimodal language models. It provides a comprehensive suite for vision-language pretraining, enabling the development of models that map images and text into shared latent spaces. The framework is distinguished by its capabilities in high-fidelity image generation and multimodal research, utilizing normalizing flows and variational autoencoders to produce images from text prompts or class labels. It supports the development of both generative and contrastive models, allowing for a wide

SegFormer is a semantic segmentation framework and transformer-based model designed for pixel-level image classification. It provides a deep learning architecture that assigns class labels to pixels using a hierarchical transformer encoder and a multi-layer perceptron decoder. The framework utilizes a hierarchical transformer encoder to process multi-scale features through a pyramid of blocks and an all-MLP decoder to aggregate these features without complex attention mechanisms. It incorporates overlap patch embedding to preserve local continuity and sequential self-attention reduction to ma

manga-ocr is a Japanese OCR engine and text extraction tool designed to recognize vertical and horizontal Japanese text from manga images. It operates as a vision encoder-decoder model that converts visual text into digital characters. The project includes an OCR training pipeline and a synthetic data generator. These tools create artificial image-text pairs by overlaying diverse Japanese text fonts onto background images to refine recognition models. The system provides automation for extracting text by monitoring the system clipboard or directories. This allows for the conversion of manga