Open-source alternatives to ScienceQA

This project is a private document analysis tool that enables conversational interaction with PDF files by executing all language model inference and processing entirely on the local machine. By running models directly within the browser or local environment, it ensures that sensitive user data remains offline and inaccessible to external servers or third-party cloud providers. The system utilizes retrieval augmented generation to provide context-aware answers, supported by local document text extraction and vector embedding indexing. This architecture allows for semantic search and informati

This project serves as an educational resource and guide for prompt engineering, providing a structured methodology for interacting with large language models. It focuses on teaching core strategies to improve the reliability, accuracy, and consistency of model outputs across a variety of natural language processing tasks. The framework emphasizes the use of standardized templates and logical decomposition to manage complex instructions. By implementing techniques such as few-shot context injection, iterative refinement, and delimiter-based segmentation, the project demonstrates how to guide

Project Page | Paper

🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.

This project is a multimodal large language model reasoning framework designed to train and evaluate models in performing chain-of-thought reasoning across text and image data. It provides a reasoning engine and training system that enable vision-language models to generate step-by-step logical rationales and final answers for complex queries. The framework utilizes a two-stage training pipeline that decouples the generation of logical justifications from final answer inference. It transforms visual data into descriptive text through image captioning and uses vision-transformer feature extrac

Shikra : Unleashing Multimodal LLM’s Referential Dialogue Magic

NeurIPS 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models

Caption-Anything is a versatile tool combining image segmentation, visual captioning, and ChatGPT, generating tailored captions with diverse controls for user preferences. https://huggingface.co/spaces/TencentARC/Caption-Anything https://huggingface.co/spaces/VIPLab/Caption-Anything

AINL 2023 IMAD: IMage Augmented multi-modal Dialogue

ACL 2024 🔥 Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.

CVPR 2024 Official Code for the Paper "Compositional Chain-of-Thought Prompting for Large Multimodal Models"

Neurips'24 Spotlight Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

CVPR2025 Highlight Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

SciGraphQA: Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs