Awesome GitHub RepositoriesVisual-to-Text Generation

Transformers.js is a JavaScript library and web machine learning framework designed to run pretrained transformer models directly in the browser. It serves as a client-side inference engine and a wrapper for the ONNX Runtime, enabling the execution of multimodal AI tasks on user devices without the need for a backend server. The library distinguishes itself by providing a unified toolkit for processing text, image, and audio data locally. This architecture supports privacy-preserving model inference and reduces latency by performing all computations on the client's hardware. Its capabilities

Open CLIP is an open source framework for training and deploying Contrastive Language-Image Pre-training models. It serves as a vision-language training framework and multimodal embedding engine that maps images and text into a shared vector space for similarity searches and zero-shot classification. The project provides a toolkit for distributed training of contrastive models and includes an image-to-text generative model for producing natural language descriptions. It supports custom text encoder integration and utilizes teacher-student model distillation to transfer knowledge from large pr

Omniparse is a multimodal content parser and generative AI ingestion engine designed to convert documents, images, and multimedia into a uniform format. It functions as a data preprocessing pipeline that transforms diverse raw data sources into structured markdown to improve the performance of large language model workflows. The system extracts text and structural data from PDFs, images, audio, and video files. It includes a web crawler that converts dynamic website content into clean markdown and a multimodal transformation process that maps disparate input formats into a unified data schema

This project is a vision language model framework and vision-to-text pipeline designed for deploying and optimizing models that process both images and text. It provides an on-device inference engine and a vision language model framework to run quantized models locally on mobile and desktop hardware accelerators. The framework features a model quantization toolkit to reduce weight precision for lower memory footprints and increased execution speed on specialized silicon. It also includes an efficient vision encoder utilizing a hybrid encoding system to compress image tokens, which reduces pro

LLaVA-NeXT es un framework de modelo de lenguaje grande multimodal y toolkit de entrenamiento diseñado para procesar imágenes intercaladas y secuencias de vídeo para generar texto. Funciona como un modelo de lenguaje visual que combina codificadores de visión con modelos de lenguaje para realizar razonamiento complejo, respuesta a preguntas y comprensión de vídeo. El sistema es capaz de analizar imágenes de alta resolución y fotogramas de vídeo temporales para describir eventos, resumir acciones y razonar a través de múltiples entradas visuales. Soporta la interpretación de documentos y gráficos, análisis de entorno espacial y la generación de subtítulos descriptivos tanto para imágenes como para vídeo. El framework incluye herramientas para ajustar modelos multimodales mediante optimización de preferencias para reducir alucinaciones y mejorar la precisión. También proporciona un servidor de inferencia para desplegar estas capacidades como un servicio API vía un backend HTTP.

LLaDA is a masked diffusion language model and conditional text generator. It generates text by iteratively refining masked tokens through a diffusion process rather than predicting the next token in a sequence. The project functions as a vision-language diffusion model, converting visual inputs into text responses. It also serves as a preference optimization framework that uses log-likelihood estimation and evidence lower bounds to tune model responses. The system supports multi-round conversational AI and text sequence evaluation. It integrates vision-language embedding for cross-modal con