Awesome GitHub RepositoriesEncoder-Combiner Architectures

Ludwig is a multimodal machine learning platform and low-code framework designed for building, training, and deploying neural networks. It enables the construction of models that process text, images, audio, and tabular data through a unified interface using declarative configuration files rather than custom code. The system features a specialized low-code framework for large language models, supporting supervised fine-tuning, preference alignment, and a constrained decoding tool to force structured data output via logit extraction. It also includes an automated model architecture search to i

Ludwig is a declarative machine learning framework designed for training neural networks and large language models using configuration files instead of manual coding. It functions as a multimodal model builder and a low-code tool for supervised fine-tuning, allowing users to build models that process mixed inputs of text, images, audio, and tabular data. The project distinguishes itself through an automated hyperparameter optimizer and a system for large language model fine-tuning using parameter-efficient adapters. It features a multimodal data pipeline and the ability to automatically gener

Cosmos is an open platform of world models, datasets, and tools for building physical AI systems such as robots and autonomous vehicles. It provides video generation and video understanding models that can generate synthetic videos and world simulations from text, image, video, or action inputs, and analyze videos to produce captions, event timestamps, spatial bounding boxes, and next-action predictions. The platform includes a world simulation generator that produces images, videos, synchronized audio, and action-conditioned rollouts for synthetic data, alongside a visual content analyzer th

AutoGluon is an automated machine learning framework designed to optimize model selection and hyperparameter tuning across tabular, text, image, and time series data. It functions as an ensemble learning library and a tabular data prediction engine, aiming to build high-accuracy predictive models without manual algorithm selection. The framework integrates multimodal machine learning pipelines that combine disparate data types into a single representation using specialized encoders. It also includes a probabilistic time series forecaster that fits multiple statistical and deep learning models

ImageBind is a multi-modal embedding model and joint representation learner that maps images, text, audio, and other modalities into a single shared vector space. It functions as a cross-modal retrieval framework designed to bind multiple sensory inputs into one cohesive mathematical embedding. The system uses a contrastive learning architecture to align disparate data types by maximizing the similarity between related samples. This allows the model to perform zero-shot multimodal classification and execute cross-modal data retrieval, such as locating visual content via natural language descr

DUSt3R is a geometric vision transformer model that predicts dense 3D pointmaps directly from one or more uncalibrated images, without requiring prior camera intrinsics, extrinsics, or known camera positions. Its core identity is an end-to-end approach to 3D reconstruction that bypasses traditional depth estimation and camera calibration pipelines, instead outputting metric-scale 3D coordinates from RGB inputs. The model processes image pairs through a shared dual-image encoder architecture, using cross-attention feature fusion in the decoder to merge features from two images into a unified p

System-bus-radio is a software-defined radio transmitter that generates AM radio signals by modulating the electromagnetic emissions from a computer's processor and memory bus, without requiring any dedicated radio hardware or physical antennas. It functions as a CPU electromagnetic emissions tool and processor-based signal generator, enabling radio transmission through precise control of CPU instructions and memory bus operations. The project encodes musical notes as sequences of frequency and duration pairs, then synthesizes the AM radio waveform in real-time by executing a tight loop of CP

This project is a high-performance C++ and CUDA neural network library designed for fast training and inference of small networks on NVIDIA GPUs. It serves as a specialized backend for neural radiance fields and coordinate-based networks, providing a fused GPU kernel library and a hash grid encoder for transforming raw input dimensions into high-dimensional representations. The library distinguishes itself through the use of C++ template metaprogramming and fused-kernel execution, which merge neural network layers into single GPU device functions to eliminate memory bottlenecks. It leverages

VisualGLM-6B is a multimodal large language model and vision-language system designed to process and generate text based on combined textual and visual inputs. It functions as a bilingual conversational AI capable of maintaining natural language interactions in both English and Chinese. The project utilizes quantized model weights to reduce memory requirements, enabling the deployment of the neural network on consumer-grade hardware. These compressed parameters allow for lower VRAM usage while maintaining the model's ability to analyze visual content and generate corresponding natural languag

Open Flamingo ist ein multimodales Large-Language-Model-Trainingsframework, das darauf ausgelegt ist, vortrainierte Vision-Encoder mit Sprachmodellen zu integrieren. Es implementiert eine Vision-Language-Architektur, die Cross-Attention-Layer nutzt, um verschachtelte Sequenzen von Bildern und Text zu verarbeiten. Das System zeichnet sich durch seine Few-Shot-multimodalen Lernfähigkeiten aus, die es dem Modell ermöglichen, sich mit einer kleinen Menge an Bild-Text-Beispielen im Prompt an neue visuelle Aufgaben anzupassen. Es unterstützt In-Context-Learning und multimodale Textgenerierung für Aufgaben wie visuelle Fragenbeantwortung (VQA) und Captioning. Das Framework enthält einen verteilten Modell-Trainer, der Datenparallelität und Gradient-Checkpointing zur Speicheroptimierung über mehrere GPUs hinweg einsetzt. Es bietet zudem Utilities für das Laden geshardeter multimodaler Datensätze, parallelisierte Modellevaluierung und Infrastruktur zum Hosten großskaliger Modelle für die Inferenz.