Awesome GitHub RepositoriesVisual Question Answering

This project is a systematic framework for English language acquisition that applies structured workflows and cognitive strategies to build linguistic proficiency. It focuses on the construction of a linguistic knowledge base, enabling learners to master vocabulary and grammar through methodical training. The methodology is distinguished by its use of computer science concepts, such as mental-model-based learning and memory buffers, to organize progression. It emphasizes a cognitive-translation bypass to develop target language thinking, reducing mental latency by processing information direc

LAVIS is a multimodal large language model framework and vision-language model library. It provides tools for training and evaluating models that integrate visual, textual, and audio data, serving as a cross-modal feature extractor and a zero-shot visual reasoning engine. The framework distinguishes itself by using frozen-backbone integration, where pretrained encoders remain non-trainable while lightweight adapter layers are updated. It employs cross-modal feature alignment to map different representations into a shared embedding space and utilizes a modular model wrapper to swap vision and

Moondream is a small-scale vision language model designed to reason across images to generate captions and answer natural language questions. It functions as an edge-optimized system capable of performing visual question answering, image captioning, and object detection. The project distinguishes itself through a lightweight architecture designed for local inference on embedded devices, workstations, and air-gapped hardware. It supports the execution of models on local GPUs and Apple Silicon to ensure data privacy and low latency. The system's capabilities include identifying precise object

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

This project is a vision language model framework and vision-to-text pipeline designed for deploying and optimizing models that process both images and text. It provides an on-device inference engine and a vision language model framework to run quantized models locally on mobile and desktop hardware accelerators. The framework features a model quantization toolkit to reduce weight precision for lower memory footprints and increased execution speed on specialized silicon. It also includes an efficient vision encoder utilizing a hybrid encoding system to compress image tokens, which reduces pro

CogVLM is a multimodal large language model designed to integrate visual and textual data for reasoning about images and generating natural language. It functions as a visual question answering system that analyzes image content to provide detailed descriptions or answer specific questions. The project includes a visual grounding model capable of mapping text descriptions to precise bounding box coordinates within an image. It also features a vision-based automation agent that analyzes screen captures to generate execution plans and interaction coordinates for software interfaces. The system

Donut is an OCR-free document transformer and end-to-end document parser. It functions as a neural network that converts unstructured document images directly into structured data or text without the use of an external optical character recognition engine. The project includes a synthetic document generator to create artificial images and ground-truth labels for training. It employs a transformer model to perform visual question answering and document image classification based on visual layout and text. The system covers several document understanding capabilities, including structured info

Genkit is an open-source framework for building AI-powered applications. It provides a unified interface for connecting to hundreds of generative AI models from multiple providers, enabling text, image, audio, and video generation through a single API. The framework structures multi-step AI interactions—including chat, retrieval-augmented generation, tool use, and agentic workflows—as composable, traceable flows with built-in streaming and state management. The framework distinguishes itself through a comprehensive developer toolkit that includes a command-line interface and a local developer

Everywhere is a desktop AI assistant that understands whatever is on your screen and can act across applications without requiring screenshots or manual context switching. It reads structured UI data through accessibility and automation APIs to perceive the active application and visible content, then provides context-aware help, summaries, translations, and answers to natural language questions about what you are viewing. The tool distinguishes itself by combining on-screen content analysis with a multi-LLM agent platform that routes requests to providers like OpenAI, Anthropic, and local mo

MMF is a modular framework for building, training, and evaluating vision-and-language models. It provides a configuration-driven experiment system where model, dataset, and training parameters are defined through composable YAML files, alongside a curated model zoo of pretrained checkpoints for state-of-the-art multimodal architectures. The framework includes a multimodal dataset loader that downloads, processes, and batches vision-and-language data, and a vision-language model trainer supporting distributed training, mixed precision, and checkpoint-based resumption. The framework distinguish

BLIP is a vision-language model framework that combines contrastive, matching, and language modeling objectives to align images with text. Built on a multimodal encoder-decoder architecture, it supports distributed data-parallel training with cosine learning rate scheduling and sliding-window metric tracking for training stability. The framework provides capabilities for image captioning, visual question answering, and cross-modal retrieval, scoring semantic alignment between images and text through learned embeddings. It includes toolkits for fine-tuning pre-trained models on custom datasets

DeepSeek-VL2 es un modelo de lenguaje grande multimodal y sistema de visión-lenguaje diseñado para analizar escenas visuales y generar texto descriptivo. Funciona como un modelo de respuesta a preguntas visuales y fundamentación visual (visual grounding), capaz de extraer información de documentos y localizar objetos o regiones específicas dentro de imágenes basadas en descripciones textuales. El proyecto utiliza una arquitectura de mezcla de expertos (mixture-of-experts) para procesar entradas combinadas de imagen y texto. Está optimizado para la inferencia mediante prellenado incremental, lo que reduce los requisitos de memoria de GPU en el hardware. El modelo cubre el análisis de datos multimodal y la comprensión de documentos visuales, incluyendo la interpretación de gráficos y diseños. Realiza inferencia visual y fundamentación para hacer coincidir consultas textuales con el contenido visual correspondiente.

nanoVLM es un framework de entrenamiento y kit de herramientas para modelos pequeños de visión-lenguaje. Proporciona un entorno basado en PyTorch para entrenar y ajustar modelos con el fin de asociar entradas de imagen con descripciones textuales y generar respuestas en lenguaje natural. El proyecto incluye una herramienta de versionado de modelos en la nube para guardar y cargar pesos de modelos en repositorios centralizados, sincronizando activos entre entornos. También cuenta con una suite de evaluación dedicada para medir la precisión y fiabilidad de los modelos de visión-lenguaje frente a datasets de tareas estándar. El framework cubre la planificación de recursos de GPU mediante la medición del consumo de VRAM y gestiona la estabilidad del entrenamiento con persistencia de estado basada en checkpoints y gestión de memoria por lotes.

Youtu Agent is an open-source framework for building, running, and evaluating autonomous agents powered by large language models. It provides the core infrastructure for creating agents that follow reasoning loops, use toolkits, and coordinate with other agents to solve complex tasks, all managed through YAML-driven configuration files. The framework distinguishes itself through its support for multi-agent orchestration, where a planner agent decomposes tasks and coordinates specialized worker agents, and through its integration with the Model Context Protocol for connecting to external toolk

Kimi-Audio is a large language model audio foundation model designed to understand audio input and generate high-fidelity speech responses in real time. It functions as a unified system encompassing a text-to-speech synthesis engine and a speech-to-text transcription tool. The project enables real-time audio conversations through a multi-modal conversation loop and chunk-wise streaming detokenization to reduce playback latency. It provides controls over speech speed, accent, and emotional tone during conversational audio generation. The system covers audio intelligence capabilities, includin

DeepSeek-VL es un modelo de lenguaje grande multimodal y motor de razonamiento de imagen a texto. Funciona como un modelo de visión-lenguaje y sistema de respuesta a preguntas visuales que integra la percepción visual con el razonamiento lingüístico para comprender y describir imágenes. El proyecto permite la comprensión multimodal de imágenes y el análisis de imágenes de documentos, procesando específicamente capturas de pantalla de páginas web y diagramas técnicos. Proporciona capacidades para IA conversacional visual, permitiendo a los usuarios interactuar con datos visuales para extraer información y realizar razonamientos complejos a través de diferentes tipos de información visual. El sistema utiliza una arquitectura transformer de visión-lenguaje que combina un Vision Transformer para la codificación visual con un modelo de lenguaje grande. Emplea ajuste de instrucciones multimodal y una capa de proyección para alinear los vectores de características visuales con el espacio de embedding del modelo de lenguaje para la generación de texto autorregresiva.

Spark NLP es un kit de herramientas para el análisis de texto escalable y aprendizaje automático construido sobre el framework de computación distribuida Apache Spark. Proporciona un framework de aprendizaje automático multimodal y un sistema de tuberías distribuido para secuenciar anotadores para procesar datos lingüísticos a gran escala. La librería incluye un procesador de texto transformer para generar embeddings vectoriales contextuales y un motor de inferencia dedicado para gestionar grandes modelos de lenguaje. El proyecto se distingue por su capacidad para procesar tipos de datos heterogéneos, incluyendo texto, audio e imágenes, dentro de una arquitectura unificada de visión-lenguaje. Admite capacidades avanzadas de IA generativa como prompt engineering, extracción de entidades estructuradas con salida JSON restringida e inferencia local para eliminar la latencia de red. Además, proporciona herramientas para la traducción entre idiomas y la clasificación zero-shot a través de modalidades de texto e imagen. El framework cubre una amplia gama de capacidades, incluyendo el entrenamiento de modelos supervisados para el reconocimiento de entidades y el análisis de sentimientos, así como la respuesta a preguntas extractiva y el resumen de documentos. Integra soporte para bases de datos vectoriales para la búsqueda de similitud y ofrece infraestructura para la aceleración por GPU y la gestión del ciclo de vida del modelo a través de un registro centralizado. El kit de herramientas permite la distribución de modelos y tuberías personalizados a través de un repositorio público y admite el despliegue de modelos mediante APIs REST.

SmolLM is a project dedicated to the development of small language models. It focuses on training and fine-tuning compact models that maintain high performance while utilizing fewer parameters. The project emphasizes efficient AI inference and on-device text generation, aiming to enable the deployment of lightweight models on edge devices with limited memory and processing power. It utilizes synthetic data generation to produce artificial datasets that improve the reasoning and training of these AI systems. The system supports a variety of optimization and training capabilities, including we

ComfyUI-nunchaku is a 4-bit diffusion inference engine and a set of nodes for running low-precision quantized diffusion models within ComfyUI visual workflows. It provides a backend that reduces memory overhead and increases generation speed for transformer models. The project includes specialized tools for identity-preserving generation and an image-to-image guidance toolkit that uses depth maps and reference images. It also features a multimodal visual question answering implementation and a utility for merging multiple quantized model files into single unified files. The engine covers a b