Awesome GitHub RepositoriesVisual-to-Text Generation

Transformers.js is a JavaScript library and web machine learning framework designed to run pretrained transformer models directly in the browser. It serves as a client-side inference engine and a wrapper for the ONNX Runtime, enabling the execution of multimodal AI tasks on user devices without the need for a backend server. The library distinguishes itself by providing a unified toolkit for processing text, image, and audio data locally. This architecture supports privacy-preserving model inference and reduces latency by performing all computations on the client's hardware. Its capabilities

Open CLIP is an open source framework for training and deploying Contrastive Language-Image Pre-training models. It serves as a vision-language training framework and multimodal embedding engine that maps images and text into a shared vector space for similarity searches and zero-shot classification. The project provides a toolkit for distributed training of contrastive models and includes an image-to-text generative model for producing natural language descriptions. It supports custom text encoder integration and utilizes teacher-student model distillation to transfer knowledge from large pr

Omniparse is a multimodal content parser and generative AI ingestion engine designed to convert documents, images, and multimedia into a uniform format. It functions as a data preprocessing pipeline that transforms diverse raw data sources into structured markdown to improve the performance of large language model workflows. The system extracts text and structural data from PDFs, images, audio, and video files. It includes a web crawler that converts dynamic website content into clean markdown and a multimodal transformation process that maps disparate input formats into a unified data schema

This project is a vision language model framework and vision-to-text pipeline designed for deploying and optimizing models that process both images and text. It provides an on-device inference engine and a vision language model framework to run quantized models locally on mobile and desktop hardware accelerators. The framework features a model quantization toolkit to reduce weight precision for lower memory footprints and increased execution speed on specialized silicon. It also includes an efficient vision encoder utilizing a hybrid encoding system to compress image tokens, which reduces pro

LLaVA-NeXT est un framework de modèle de langage multimodal et une boîte à outils d'entraînement conçus pour traiter des séquences entrelacées d'images et de vidéos afin de générer du texte. Il fonctionne comme un modèle de langage visuel qui combine des encodeurs de vision avec des modèles de langage pour effectuer des raisonnements complexes, répondre à des questions et comprendre la vidéo. Le système est capable d'analyser des images haute résolution et des trames vidéo temporelles pour décrire des événements, résumer des actions et raisonner à travers plusieurs entrées visuelles. Il prend en charge l'interprétation de documents et de graphiques, l'analyse de l'environnement spatial et la génération de légendes descriptives pour les images et les vidéos. Le framework inclut des outils pour ajuster les modèles multimodaux via l'optimisation des préférences afin de réduire les hallucinations et améliorer la précision. Il fournit également un serveur d'inférence pour déployer ces capacités en tant que service API via un backend HTTP.

LLaDA is a masked diffusion language model and conditional text generator. It generates text by iteratively refining masked tokens through a diffusion process rather than predicting the next token in a sequence. The project functions as a vision-language diffusion model, converting visual inputs into text responses. It also serves as a preference optimization framework that uses log-likelihood estimation and evidence lower bounds to tune model responses. The system supports multi-round conversational AI and text sequence evaluation. It integrates vision-language embedding for cross-modal con