# evolvinglmms-lab/otter

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/evolvinglmms-lab-otter).**

3,331 stars · 208 forks · Python · mit

## Links

- GitHub: https://github.com/EvolvingLMMs-Lab/Otter
- Homepage: https://otter-ntu.github.io/
- awesome-repositories: https://awesome-repositories.com/repository/evolvinglmms-lab-otter.md

## Topics

`artificial-inteligence` `chatgpt` `deep-learning` `embodied-ai` `foundation-models` `gpt-4` `instruction-tuning` `large-scale-models` `machine-learning` `multi-modality` `visual-language-learning`

## Description

Otter is a framework and toolkit for the pretraining, fine-tuning, and evaluation of vision-language models. It provides a pipeline for training large language models to process high-resolution images and video frames, integrating visual encoders with textual token spaces.

The system is designed for multi-visual input processing, allowing models to interpret multiple images or video sequences within a single prompt. It supports multi-round conversation management to maintain context across interactions for detailed scene comprehension and visual reasoning.

The framework covers a full development lifecycle, including foundational pretraining, supervised fine-tuning, and visual instruction tuning. It also includes a dedicated evaluation suite to measure reasoning accuracy and performance when processing combined visual and textual data.

## Tags

### Artificial Intelligence & ML

- [Training Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/model-training-frameworks/vision-model-training/vision-language-training/training-frameworks.md) — Provides a comprehensive framework for pretraining and fine-tuning vision-language models to process high-resolution images and video.
- [Scene Comprehension](https://awesome-repositories.com/f/artificial-intelligence-ml/agent-architectures/orchestration-engines/ai-agent/reasoning-action-loops/visual-reasoning/scene-comprehension.md) — Enables detailed scene comprehension by analyzing multiple images or video sequences within a single conversation.
- [Visual-Textual Alignments](https://awesome-repositories.com/f/artificial-intelligence-ml/cross-modal-representations/visual-textual-alignments.md) — Maps visual encoder embeddings into the textual token space using a learned projection layer for unified multimodal processing.
- [Instruction Tuning Pipelines](https://awesome-repositories.com/f/artificial-intelligence-ml/instruction-tuning-pipelines.md) — Ships automated workflows to format high-resolution image and text pairs into conversational templates for supervised fine-tuning.
- [Multimodal](https://awesome-repositories.com/f/artificial-intelligence-ml/instruction-tuning/multimodal.md) — Performs in-context instruction tuning to help models interpret high-resolution images, videos, and text using provided examples. ([source](https://cdn.jsdelivr.net/gh/evolvinglmms-lab/otter@main/README.md))
- [Multimodal Fine-Tuning](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/fine-tuning-and-customization/model-fine-tuning/multimodal-fine-tuning.md) — Adapts pretrained multimodal models to specific downstream tasks to improve targeted visual and text interpretations. ([source](https://otter-ntu.github.io/))
- [Multimodal Encoders](https://awesome-repositories.com/f/artificial-intelligence-ml/multimodal-encoders.md) — Integrates pretrained visual backbones with large language models to interpret spatial information as semantic tokens.
- [Vision-Language Pipelines](https://awesome-repositories.com/f/artificial-intelligence-ml/vision-language-pipelines.md) — Provides end-to-end workflows for training vision-language models that maintain context across multi-turn conversations.
- [Visual Instruction Tuning](https://awesome-repositories.com/f/artificial-intelligence-ml/visual-instruction-tuning.md) — Offers a toolkit for adapting pretrained models to follow complex visual and textual instructions.
- [Multi-turn Interaction Managers](https://awesome-repositories.com/f/artificial-intelligence-ml/agentic-systems-frameworks/conversational-voice-interaction/conversational-ai-agents/conversational-turn-detection/multi-turn-interaction-managers.md) — Manages stateful, multi-round interactions to maintain context for complex visual reasoning and scene comprehension. ([source](https://cdn.jsdelivr.net/gh/evolvinglmms-lab/otter@main/README.md))
- [Cross-Modal Context Management](https://awesome-repositories.com/f/artificial-intelligence-ml/cross-modal-context-management.md) — Provides a shared memory buffer of image tokens and text embeddings to enable reasoning across multiple visual inputs.
- [In-Context Learning Engines](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/architectures/sequence-models/multi-task-learning-models/in-context-learning-engines.md) — Implements few-shot demonstration pairs within prompts to guide model output formats without updating parameters.
- [Representative Frame Sampling](https://awesome-repositories.com/f/artificial-intelligence-ml/sequence-modeling/temporal-sequence-processors/representative-frame-sampling.md) — Processes video content by extracting discrete representative frames to treat temporal data as a series of visual inputs.

### Part of an Awesome List

- [Multimodal Pretraining](https://awesome-repositories.com/f/awesome-lists/ai/multimodal-pretraining.md) — Trains models on specialized datasets to establish foundational understanding of high-resolution visual and textual inputs. ([source](https://otter-ntu.github.io/))

### Graphics & Multimedia

- [Multi-Visual Context Processing](https://awesome-repositories.com/f/graphics-multimedia/image-editing-processing/image-processing/multi-image-sample-processing/multi-visual-context-processing.md) — Interprets multiple images or video frames within a single prompt to follow instructions spanning different visual contexts. ([source](https://cdn.jsdelivr.net/gh/evolvinglmms-lab/otter@main/README.md))

### Testing & Quality Assurance

- [Multimodal Reasoning Evaluations](https://awesome-repositories.com/f/testing-quality-assurance/model-testing/model-evaluation/multimodal-reasoning-evaluations.md) — Measures reasoning accuracy and performance when processing combined visual and textual data.