How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
VLMEvalKit is a vision-language model evaluation framework and inference engine designed to run standardized benchmarks and measure model accuracy across diverse visual datasets. It serves as a multimodal model benchmark and performance toolkit for calculating metrics and comparing model responses. The toolkit includes a specialized visual reasoning evaluator that uses adversarial samples to distinguish actual image understanding from reliance on language patterns. It also provides capabilities for image generation evaluation, testing a model's ability to create or modify visuals based on tex
✨✨CVPR 2025 Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
paper: https://arxiv.org/abs/2307.02469 page: https://lynx-llm.github.io/
An automatic evaluation framework for Multimodal Chain-of-Thought.
[NAACL 2024] MMC: Advancing Multimodal Chart Understanding with LLM Instruction Tuning
The main features of fuxiaoliu/mmc are: Multimodal Benchmarks.
Projects with overlapping indexed features include: open-compass/vlmevalkit — VLMEvalKit is a vision-language model evaluation framework and inference engine designed to run standardized… bradyfu/video-mme — ✨✨[CVPR 2025] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. bytedance/lynx-llm — paper: https://arxiv.org/abs/2307.02469 page: https://lynx-llm.github.io/. cmmmu-benchmark/cmmmu — 🌐 Homepage | 🤗 Paper | 📖 arXiv | 🤗 Dataset | 🏆 EvalAI | GitHub. damo-nlp-sg/m3exam — Data and code for paper "M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models". alenai97/miceval — An automatic evaluation framework for Multimodal Chain-of-Thought.