awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
EvolvingLMMs-Lab avatar

EvolvingLMMs-Lab/Otter

0
View on GitHub↗

Otter

Otter is a framework and toolkit for the pretraining, fine-tuning, and evaluation of vision-language models. It provides a pipeline for training large language models to process high-resolution images and video frames, integrating visual encoders with textual token spaces.

The system is designed for multi-visual input processing, allowing models to interpret multiple images or video sequences within a single prompt. It supports multi-round conversation management to maintain context across interactions for detailed scene comprehension and visual reasoning.

The framework covers a full development lifecycle, including foundational pretraining, supervised fine-tuning, and visual instruction tuning. It also includes a dedicated evaluation suite to measure reasoning accuracy and performance when processing combined visual and textual data.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI
otter-ntu.github.io
↗

Features

  • Training Frameworks - Provides a comprehensive framework for pretraining and fine-tuning vision-language models to process high-resolution images and video.
  • Scene Comprehension - Enables detailed scene comprehension by analyzing multiple images or video sequences within a single conversation.
  • Visual-Textual Alignments - Maps visual encoder embeddings into the textual token space using a learned projection layer for unified multimodal processing.
  • Instruction Tuning Pipelines - Ships automated workflows to format high-resolution image and text pairs into conversational templates for supervised fine-tuning.
  • Multimodal - Performs in-context instruction tuning to help models interpret high-resolution images, videos, and text using provided examples.
  • Multimodal Fine-Tuning - Adapts pretrained multimodal models to specific downstream tasks to improve targeted visual and text interpretations.
  • Multimodal Encoders - Integrates pretrained visual backbones with large language models to interpret spatial information as semantic tokens.
  • Vision-Language Pipelines - Provides end-to-end workflows for training vision-language models that maintain context across multi-turn conversations.
  • Visual Instruction Tuning - Offers a toolkit for adapting pretrained models to follow complex visual and textual instructions.
  • Multimodal Pretraining - Trains models on specialized datasets to establish foundational understanding of high-resolution visual and textual inputs.
  • Multi-Visual Context Processing - Interprets multiple images or video frames within a single prompt to follow instructions spanning different visual contexts.
  • Multimodal Reasoning Evaluations - Measures reasoning accuracy and performance when processing combined visual and textual data.
  • Multi-turn Interaction Managers - Manages stateful, multi-round interactions to maintain context for complex visual reasoning and scene comprehension.
  • Cross-Modal Context Management - Provides a shared memory buffer of image tokens and text embeddings to enable reasoning across multiple visual inputs.
  • In-Context Learning Engines - Implements few-shot demonstration pairs within prompts to guide model output formats without updating parameters.
  • Representative Frame Sampling - Processes video content by extracting discrete representative frames to treat temporal data as a series of visual inputs.
3,331 stars·208 forks·Python·mit·21 views

Star history

Star history chart for evolvinglmms-lab/otterStar history chart for evolvinglmms-lab/otter

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does evolvinglmms-lab/otter do?

Otter is a framework and toolkit for the pretraining, fine-tuning, and evaluation of vision-language models. It provides a pipeline for training large language models to process high-resolution images and video frames, integrating visual encoders with textual token spaces.

What are the main features of evolvinglmms-lab/otter?

The main features of evolvinglmms-lab/otter are: Training Frameworks, Scene Comprehension, Visual-Textual Alignments, Instruction Tuning Pipelines, Multimodal, Multimodal Fine-Tuning, Multimodal Encoders, Vision-Language Pipelines.

Which projects share features with evolvinglmms-lab/otter?

Projects with overlapping indexed features include: haotian-liu/llava — LLaVA is a multimodal large language model architecture designed to process and interpret both image and text inputs… vision-cair/minigpt-4 — MiniGPT-4 is a multimodal AI framework and large language model that integrates vision encoders with language models… mlfoundations/open_flamingo — Open Flamingo is a multimodal large language model training framework designed to integrate pretrained vision encoders… salesforce/blip — BLIP is a vision-language model framework that combines contrastive, matching, and language modeling objectives to… huggingface/nanovlm — nanoVLM is a training framework and toolkit for small vision-language models. It provides a PyTorch-based environment… jingyaogong/minimind-v.

Projects sharing features with Otter

These projects share indexed features with Otter. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • haotian-liu/llavahaotian-liu avatar

    haotian-liu/LLaVA

    24,465View on GitHub↗

    LLaVA is a multimodal large language model architecture designed to process and interpret both image and text inputs to generate natural language responses. It functions as a research-oriented platform for visual instruction tuning, providing a framework to align language models with human intent through training on diverse datasets of paired images and text queries. The system distinguishes itself through a specialized vision-language training pipeline that connects visual data to language models using projection layers and instruction-based fine-tuning. It supports distributed inference by

    Pythonchatbotchatgptfoundation-models
    View on GitHub↗24,465
  • vision-cair/minigpt-4Vision-CAIR avatar

    Vision-CAIR/MiniGPT-4

    25,679View on GitHub↗

    MiniGPT-4 is a multimodal AI framework and large language model that integrates vision encoders with language models to process and reason about combined image and text inputs. It functions as a vision-language model capable of image-based conversational AI, visual question answering, and multimodal logical reasoning. The project utilizes a pretrained vision-language integration strategy that connects a vision encoder to a language model via a linear projection layer. This approach employs frozen-backbone training to align visual representations with linguistic tokens while keeping the primar

    Python
    View on GitHub↗25,679
  • mlfoundations/open_flamingomlfoundations avatar

    mlfoundations/open_flamingo

    4,107View on GitHub↗

    Open Flamingo is a multimodal large language model training framework designed to integrate pretrained vision encoders with language models. It implements a vision-language architecture that utilizes cross-attention layers to process interleaved sequences of images and text. The system is characterized by its few-shot multimodal learning capabilities, allowing the model to adapt to new visual tasks using a small set of image-text examples provided in the prompt. It supports in-context learning and multimodal text generation for tasks such as visual question answering and captioning. The fram

    Pythoncomputer-visiondeep-learningflamingo
    View on GitHub↗4,107
  • jingyaogong/minimind-vjingyaogong avatar

    jingyaogong/minimind-v

    6,431View on GitHub↗
    Pythonartificial-intelligencechatgptvision-language-model
    View on GitHub↗6,431
Compare all 30 related projects→