Awesome GitHub RepositoriesVisual-to-Text Generation

Transformers.js is a JavaScript library and web machine learning framework designed to run pretrained transformer models directly in the browser. It serves as a client-side inference engine and a wrapper for the ONNX Runtime, enabling the execution of multimodal AI tasks on user devices without the need for a backend server. The library distinguishes itself by providing a unified toolkit for processing text, image, and audio data locally. This architecture supports privacy-preserving model inference and reduces latency by performing all computations on the client's hardware. Its capabilities

Open CLIP is an open source framework for training and deploying Contrastive Language-Image Pre-training models. It serves as a vision-language training framework and multimodal embedding engine that maps images and text into a shared vector space for similarity searches and zero-shot classification. The project provides a toolkit for distributed training of contrastive models and includes an image-to-text generative model for producing natural language descriptions. It supports custom text encoder integration and utilizes teacher-student model distillation to transfer knowledge from large pr

Omniparse is a multimodal content parser and generative AI ingestion engine designed to convert documents, images, and multimedia into a uniform format. It functions as a data preprocessing pipeline that transforms diverse raw data sources into structured markdown to improve the performance of large language model workflows. The system extracts text and structural data from PDFs, images, audio, and video files. It includes a web crawler that converts dynamic website content into clean markdown and a multimodal transformation process that maps disparate input formats into a unified data schema

This project is a vision language model framework and vision-to-text pipeline designed for deploying and optimizing models that process both images and text. It provides an on-device inference engine and a vision language model framework to run quantized models locally on mobile and desktop hardware accelerators. The framework features a model quantization toolkit to reduce weight precision for lower memory footprints and increased execution speed on specialized silicon. It also includes an efficient vision encoder utilizing a hybrid encoding system to compress image tokens, which reduces pro

LLaVA-NeXT ist ein multimodales Large-Language-Model-Framework und ein Trainings-Toolkit, das darauf ausgelegt ist, verschachtelte Bild- und Videosequenzen zu verarbeiten, um Text zu generieren. Es fungiert als visuelles Sprachmodell, das Vision-Encoder mit Sprachmodellen kombiniert, um komplexes Reasoning, Question-Answering und Videoverständnis durchzuführen. Das System ist in der Lage, hochauflösende Bilder und temporale Videoframes zu analysieren, um Ereignisse zu beschreiben, Aktionen zusammenzufassen und über mehrere visuelle Inputs hinweg zu schlussfolgern. Es unterstützt die Interpretation von Dokumenten und Diagrammen, räumliche Umgebungsanalyse sowie die Generierung beschreibender Bildunterschriften für Bilder und Videos. Das Framework enthält Tools zum Tuning multimodaler Modelle mittels Preference-Optimization, um Halluzinationen zu reduzieren und die Genauigkeit zu verbessern. Es bietet zudem einen Inference-Server, um diese Funktionen als API-Dienst via HTTP-Backend bereitzustellen.

LLaDA is a masked diffusion language model and conditional text generator. It generates text by iteratively refining masked tokens through a diffusion process rather than predicting the next token in a sequence. The project functions as a vision-language diffusion model, converting visual inputs into text responses. It also serves as a preference optimization framework that uses log-likelihood estimation and evidence lower bounds to tune model responses. The system supports multi-round conversational AI and text sequence evaluation. It integrates vision-language embedding for cross-modal con