Awesome GitHub RepositoriesVisual Question Answering

This project is a systematic framework for English language acquisition that applies structured workflows and cognitive strategies to build linguistic proficiency. It focuses on the construction of a linguistic knowledge base, enabling learners to master vocabulary and grammar through methodical training. The methodology is distinguished by its use of computer science concepts, such as mental-model-based learning and memory buffers, to organize progression. It emphasizes a cognitive-translation bypass to develop target language thinking, reducing mental latency by processing information direc

LAVIS is a multimodal large language model framework and vision-language model library. It provides tools for training and evaluating models that integrate visual, textual, and audio data, serving as a cross-modal feature extractor and a zero-shot visual reasoning engine. The framework distinguishes itself by using frozen-backbone integration, where pretrained encoders remain non-trainable while lightweight adapter layers are updated. It employs cross-modal feature alignment to map different representations into a shared embedding space and utilizes a modular model wrapper to swap vision and

Moondream is a small-scale vision language model designed to reason across images to generate captions and answer natural language questions. It functions as an edge-optimized system capable of performing visual question answering, image captioning, and object detection. The project distinguishes itself through a lightweight architecture designed for local inference on embedded devices, workstations, and air-gapped hardware. It supports the execution of models on local GPUs and Apple Silicon to ensure data privacy and low latency. The system's capabilities include identifying precise object

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

This project is a vision language model framework and vision-to-text pipeline designed for deploying and optimizing models that process both images and text. It provides an on-device inference engine and a vision language model framework to run quantized models locally on mobile and desktop hardware accelerators. The framework features a model quantization toolkit to reduce weight precision for lower memory footprints and increased execution speed on specialized silicon. It also includes an efficient vision encoder utilizing a hybrid encoding system to compress image tokens, which reduces pro

CogVLM is a multimodal large language model designed to integrate visual and textual data for reasoning about images and generating natural language. It functions as a visual question answering system that analyzes image content to provide detailed descriptions or answer specific questions. The project includes a visual grounding model capable of mapping text descriptions to precise bounding box coordinates within an image. It also features a vision-based automation agent that analyzes screen captures to generate execution plans and interaction coordinates for software interfaces. The system

Donut is an OCR-free document transformer and end-to-end document parser. It functions as a neural network that converts unstructured document images directly into structured data or text without the use of an external optical character recognition engine. The project includes a synthetic document generator to create artificial images and ground-truth labels for training. It employs a transformer model to perform visual question answering and document image classification based on visual layout and text. The system covers several document understanding capabilities, including structured info

Genkit is an open-source framework for building AI-powered applications. It provides a unified interface for connecting to hundreds of generative AI models from multiple providers, enabling text, image, audio, and video generation through a single API. The framework structures multi-step AI interactions—including chat, retrieval-augmented generation, tool use, and agentic workflows—as composable, traceable flows with built-in streaming and state management. The framework distinguishes itself through a comprehensive developer toolkit that includes a command-line interface and a local developer

Everywhere is a desktop AI assistant that understands whatever is on your screen and can act across applications without requiring screenshots or manual context switching. It reads structured UI data through accessibility and automation APIs to perceive the active application and visible content, then provides context-aware help, summaries, translations, and answers to natural language questions about what you are viewing. The tool distinguishes itself by combining on-screen content analysis with a multi-LLM agent platform that routes requests to providers like OpenAI, Anthropic, and local mo

MMF is a modular framework for building, training, and evaluating vision-and-language models. It provides a configuration-driven experiment system where model, dataset, and training parameters are defined through composable YAML files, alongside a curated model zoo of pretrained checkpoints for state-of-the-art multimodal architectures. The framework includes a multimodal dataset loader that downloads, processes, and batches vision-and-language data, and a vision-language model trainer supporting distributed training, mixed precision, and checkpoint-based resumption. The framework distinguish

BLIP is a vision-language model framework that combines contrastive, matching, and language modeling objectives to align images with text. Built on a multimodal encoder-decoder architecture, it supports distributed data-parallel training with cosine learning rate scheduling and sliding-window metric tracking for training stability. The framework provides capabilities for image captioning, visual question answering, and cross-modal retrieval, scoring semantic alignment between images and text through learned embeddings. It includes toolkits for fine-tuning pre-trained models on custom datasets

DeepSeek-VL2 هو نموذج لغوي كبير متعدد الوسائط ونظام رؤية لغوية مصمم لتحليل المشاهد المرئية وتوليد نص وصفي. يعمل كنموذج للإجابة على الأسئلة المرئية والتأريض المرئي، وقادر على استخراج المعلومات من المستندات وتحديد كائنات أو مناطق محددة داخل الصور بناءً على أوصاف نصية. يستخدم المشروع معمارية خليط من الخبراء (mixture-of-experts) لمعالجة مدخلات الصور والنصوص المدمجة. تم تحسينه للاستدلال من خلال التعبئة التزايدية (incremental prefilling)، مما يقلل من متطلبات ذاكرة GPU على الأجهزة. يغطي النموذج تحليل البيانات متعدد الوسائط وفهم المستندات المرئية، بما في ذلك تفسير المخططات والتخطيطات. يقوم بإجراء استدلال مرئي وتأريض لمطابقة الاستعلامات النصية مع المحتوى المرئي المقابل.

nanoVLM is a training framework and toolkit for small vision-language models. It provides a PyTorch-based environment for training and fine-tuning models to associate image inputs with textual descriptions and generate natural language answers. The project includes a cloud model versioning tool for saving and loading model weights to centralized repositories to synchronize assets across environments. It also features a dedicated evaluation suite for measuring the accuracy and reliability of vision-language models against standard task datasets. The framework covers GPU resource planning thro

Youtu Agent is an open-source framework for building, running, and evaluating autonomous agents powered by large language models. It provides the core infrastructure for creating agents that follow reasoning loops, use toolkits, and coordinate with other agents to solve complex tasks, all managed through YAML-driven configuration files. The framework distinguishes itself through its support for multi-agent orchestration, where a planner agent decomposes tasks and coordinates specialized worker agents, and through its integration with the Model Context Protocol for connecting to external toolk

Kimi-Audio is a large language model audio foundation model designed to understand audio input and generate high-fidelity speech responses in real time. It functions as a unified system encompassing a text-to-speech synthesis engine and a speech-to-text transcription tool. The project enables real-time audio conversations through a multi-modal conversation loop and chunk-wise streaming detokenization to reduce playback latency. It provides controls over speech speed, accent, and emotional tone during conversational audio generation. The system covers audio intelligence capabilities, includin

DeepSeek-VL هو نموذج لغة كبير متعدد الوسائط ومحرك استدلال من الصورة إلى النص. يعمل كنموذج رؤية-لغة ونظام للإجابة على الأسئلة المرئية يدمج الإدراك البصري مع الاستدلال اللغوي لفهم ووصف الصور. يمكن المشروع من فهم الصور متعددة الوسائط وتحليل صور المستندات، وتحديداً معالجة لقطات شاشة صفحات الويب والمخططات التقنية. ويوفر إمكانيات للذكاء الاصطناعي التحادثي المرئي، مما يسمح للمستخدمين بالتفاعل مع البيانات المرئية لاستخراج الرؤى وإجراء استدلال معقد عبر أنواع مختلفة من المعلومات المرئية. يستخدم النظام بنية محول رؤية-لغة تجمع بين محول رؤية (Vision Transformer) للترميز البصري ونموذج لغة كبير. ويستخدم ضبط التعليمات متعدد الوسائط وطبقة إسقاط لمحاذاة متجهات الميزات المرئية مع مساحة تضمين نموذج اللغة لتوليد النص التلقائي.

Spark NLP هي مجموعة أدوات لتحليل النصوص القابل للتوسع والتعلم الآلي مبنية على إطار عمل الحوسبة الموزعة Apache Spark. توفر إطار عمل للتعلم الآلي متعدد الوسائط ونظام خط أنابيب موزع لتسلسل أدوات التعليق لمعالجة البيانات اللغوية على نطاق واسع. تتضمن المكتبة معالج نصوص محولاً (transformer) لتوليد تضمينات متجهات سياقية ومحرك استدلال مخصص لإدارة نماذج اللغة الكبيرة. يتميز المشروع بقدرته على معالجة أنواع البيانات غير المتجانسة، بما في ذلك النصوص والصوت والصور، ضمن بنية رؤية-لغة موحدة. ويدعم إمكانيات الذكاء الاصطناعي التوليدي المتقدمة مثل هندسة الأوامر (prompt engineering)، واستخراج الكيانات المهيكلة مع مخرجات JSON مقيدة، والاستدلال المحلي للقضاء على زمن انتقال الشبكة. بالإضافة إلى ذلك، يوفر أدوات للترجمة عبر اللغات والتصنيف بدون تدريب عبر كل من وسائط النص والصورة. يغطي إطار العمل مجموعة واسعة من الإمكانيات، بما في ذلك تدريب النماذج الخاضعة للإشراف للتعرف على الكيانات وتحليل المشاعر، بالإضافة إلى الإجابة على الأسئلة الاستخراجية وتلخيص المستندات. ويدمج دعم قاعدة بيانات المتجهات للبحث عن التشابه ويوفر بنية تحتية لتسريع GPU وإدارة دورة حياة النموذج عبر سجل مركزي. تسمح مجموعة الأدوات بتوزيع النماذج وخطوط الأنابيب المخصصة عبر مستودع عام وتدعم نشر النماذج عبر واجهات برمجة تطبيقات REST.

SmolLM is a project dedicated to the development of small language models. It focuses on training and fine-tuning compact models that maintain high performance while utilizing fewer parameters. The project emphasizes efficient AI inference and on-device text generation, aiming to enable the deployment of lightweight models on edge devices with limited memory and processing power. It utilizes synthetic data generation to produce artificial datasets that improve the reasoning and training of these AI systems. The system supports a variety of optimization and training capabilities, including we

ComfyUI-nunchaku is a 4-bit diffusion inference engine and a set of nodes for running low-precision quantized diffusion models within ComfyUI visual workflows. It provides a backend that reduces memory overhead and increases generation speed for transformer models. The project includes specialized tools for identity-preserving generation and an image-to-image guidance toolkit that uses depth maps and reference images. It also features a multimodal visual question answering implementation and a utility for merging multiple quantized model files into single unified files. The engine covers a b