Awesome GitHub RepositoriesVisual-to-Text Generation

Transformers.js is a JavaScript library and web machine learning framework designed to run pretrained transformer models directly in the browser. It serves as a client-side inference engine and a wrapper for the ONNX Runtime, enabling the execution of multimodal AI tasks on user devices without the need for a backend server. The library distinguishes itself by providing a unified toolkit for processing text, image, and audio data locally. This architecture supports privacy-preserving model inference and reduces latency by performing all computations on the client's hardware. Its capabilities

Open CLIP is an open source framework for training and deploying Contrastive Language-Image Pre-training models. It serves as a vision-language training framework and multimodal embedding engine that maps images and text into a shared vector space for similarity searches and zero-shot classification. The project provides a toolkit for distributed training of contrastive models and includes an image-to-text generative model for producing natural language descriptions. It supports custom text encoder integration and utilizes teacher-student model distillation to transfer knowledge from large pr

Omniparse is a multimodal content parser and generative AI ingestion engine designed to convert documents, images, and multimedia into a uniform format. It functions as a data preprocessing pipeline that transforms diverse raw data sources into structured markdown to improve the performance of large language model workflows. The system extracts text and structural data from PDFs, images, audio, and video files. It includes a web crawler that converts dynamic website content into clean markdown and a multimodal transformation process that maps disparate input formats into a unified data schema

This project is a vision language model framework and vision-to-text pipeline designed for deploying and optimizing models that process both images and text. It provides an on-device inference engine and a vision language model framework to run quantized models locally on mobile and desktop hardware accelerators. The framework features a model quantization toolkit to reduce weight precision for lower memory footprints and increased execution speed on specialized silicon. It also includes an efficient vision encoder utilizing a hybrid encoding system to compress image tokens, which reduces pro

LLaVA-NeXT هو إطار عمل نموذج لغة كبير متعدد الوسائط ومجموعة أدوات تدريب مصممة لمعالجة الصور المتداخلة وتسلسلات الفيديو لتوليد النص. يعمل كنموذج لغة مرئي يجمع بين مشفرات الرؤية ونماذج اللغة لإجراء تفكير معقد، والإجابة على الأسئلة، وفهم الفيديو. النظام قادر على تحليل الصور عالية الدقة وإطارات الفيديو الزمنية لوصف الأحداث، وتلخيص الإجراءات، والتفكير عبر مدخلات مرئية متعددة. يدعم تفسير المستندات والمخططات، وتحليل البيئة المكانية، وتوليد تسميات توضيحية وصفية لكل من الصور والفيديو. يتضمن إطار العمل أدوات لضبط النماذج متعددة الوسائط من خلال تحسين التفضيلات لتقليل الهلوسة وتحسين الدقة. كما يوفر خادم استدلال لنشر هذه القدرات كخدمة API عبر خلفية HTTP.

LLaDA is a masked diffusion language model and conditional text generator. It generates text by iteratively refining masked tokens through a diffusion process rather than predicting the next token in a sequence. The project functions as a vision-language diffusion model, converting visual inputs into text responses. It also serves as a preference optimization framework that uses log-likelihood estimation and evidence lower bounds to tune model responses. The system supports multi-round conversational AI and text sequence evaluation. It integrates vision-language embedding for cross-modal con