amazon-science/mm-cot
Mm Cot
mm-cot 是一个多模态语言模型推理框架，旨在训练和评估跨文本和图像数据执行思维链 (chain-of-thought) 推理的模型。它提供了用于实现逐步逻辑推理以提高预测准确性的核心系统，包括视觉语言模型训练器和多模态基准评估器。
该框架通过解耦的推理生成过程脱颖而出，将逻辑论证的训练与最终答案的推理分离开来。它利用视觉 Transformer 特征提取和图像描述生成，将原始视觉数据转换为嵌入和文本上下文，以进行后续的语言处理。
该项目涵盖了广泛的能力领域，包括多模态输入编码、推理器训练和推理执行。它还包括用于语义相似度评估和基于模板的提示格式化的工具，以确保跨标准化基准的一致推理模式。
Features
- Chain-of-Thought Reasoning Engines - Provides a logic processor that decouples the generation of step-by-step rationales from final model predictions.
- Feature Extraction - Converts raw image files into structured numerical representations to prepare visual data for model training.
- Image Captioning Models - Utilizes neural network models to generate textual descriptions of images as a foundation for reasoning.
- Vision-Language Trainers - Trains multimodal models to combine vision transformers and linguistic processing for logical answer generation.
- Vision-Language Training - Trains models to align visual and textual data to perform complex downstream reasoning tasks.
- Multimodal Rationales - Implements a system to produce step-by-step logical rationales based on combined text and image data to solve complex queries.
- Multimodal Reasoners - Provides a framework for training and evaluating large language models to perform multimodal chain-of-thought reasoning.
- Vision Transformer Encoders - Employs vision transformer encoders to convert raw visual data into structured embeddings for linguistic processing.
- Decoupled Rationale Training - Implements training processes that separate the generation of logical justifications from final answer inference.
- Reasoning Sequence Pipelines - Implements a two-stage pipeline that first develops logical rationales and then refines the final answer prediction.
- Multimodal Reasoning Evaluations - Measures model accuracy and logic using benchmarks that require combined image and text analysis.
- Complex Query Inference - Predicts answers to difficult questions by separating the generation of logical justifications from the final output.
- Multimodal Inference Pipelines - Processes multimodal data through trained reasoning models to predict answers for complex queries.
- Prompt Templates - Uses standardized prompt templates to ensure consistent reasoning patterns across multimodal benchmarks.
- Output Similarity Evaluators - Provides metrics to quantify the factual alignment and similarity between generated rationales and reference data.
- Chain Of Thought - Multimodal chain-of-thought reasoning for language models.
- Reasoning And Planning - Multimodal chain-of-thought reasoning for complex inputs.
Star 历史
