Awesome GitHub RepositoriesEncoder-Combiner Architectures

Ludwig is a multimodal machine learning platform and low-code framework designed for building, training, and deploying neural networks. It enables the construction of models that process text, images, audio, and tabular data through a unified interface using declarative configuration files rather than custom code. The system features a specialized low-code framework for large language models, supporting supervised fine-tuning, preference alignment, and a constrained decoding tool to force structured data output via logit extraction. It also includes an automated model architecture search to i

Ludwig is a declarative machine learning framework designed for training neural networks and large language models using configuration files instead of manual coding. It functions as a multimodal model builder and a low-code tool for supervised fine-tuning, allowing users to build models that process mixed inputs of text, images, audio, and tabular data. The project distinguishes itself through an automated hyperparameter optimizer and a system for large language model fine-tuning using parameter-efficient adapters. It features a multimodal data pipeline and the ability to automatically gener

Cosmos is an open platform of world models, datasets, and tools for building physical AI systems such as robots and autonomous vehicles. It provides video generation and video understanding models that can generate synthetic videos and world simulations from text, image, video, or action inputs, and analyze videos to produce captions, event timestamps, spatial bounding boxes, and next-action predictions. The platform includes a world simulation generator that produces images, videos, synchronized audio, and action-conditioned rollouts for synthetic data, alongside a visual content analyzer th

AutoGluon is an automated machine learning framework designed to optimize model selection and hyperparameter tuning across tabular, text, image, and time series data. It functions as an ensemble learning library and a tabular data prediction engine, aiming to build high-accuracy predictive models without manual algorithm selection. The framework integrates multimodal machine learning pipelines that combine disparate data types into a single representation using specialized encoders. It also includes a probabilistic time series forecaster that fits multiple statistical and deep learning models

ImageBind is a multi-modal embedding model and joint representation learner that maps images, text, audio, and other modalities into a single shared vector space. It functions as a cross-modal retrieval framework designed to bind multiple sensory inputs into one cohesive mathematical embedding. The system uses a contrastive learning architecture to align disparate data types by maximizing the similarity between related samples. This allows the model to perform zero-shot multimodal classification and execute cross-modal data retrieval, such as locating visual content via natural language descr

DUSt3R is a geometric vision transformer model that predicts dense 3D pointmaps directly from one or more uncalibrated images, without requiring prior camera intrinsics, extrinsics, or known camera positions. Its core identity is an end-to-end approach to 3D reconstruction that bypasses traditional depth estimation and camera calibration pipelines, instead outputting metric-scale 3D coordinates from RGB inputs. The model processes image pairs through a shared dual-image encoder architecture, using cross-attention feature fusion in the decoder to merge features from two images into a unified p

System-bus-radio is a software-defined radio transmitter that generates AM radio signals by modulating the electromagnetic emissions from a computer's processor and memory bus, without requiring any dedicated radio hardware or physical antennas. It functions as a CPU electromagnetic emissions tool and processor-based signal generator, enabling radio transmission through precise control of CPU instructions and memory bus operations. The project encodes musical notes as sequences of frequency and duration pairs, then synthesizes the AM radio waveform in real-time by executing a tight loop of CP

This project is a high-performance C++ and CUDA neural network library designed for fast training and inference of small networks on NVIDIA GPUs. It serves as a specialized backend for neural radiance fields and coordinate-based networks, providing a fused GPU kernel library and a hash grid encoder for transforming raw input dimensions into high-dimensional representations. The library distinguishes itself through the use of C++ template metaprogramming and fused-kernel execution, which merge neural network layers into single GPU device functions to eliminate memory bottlenecks. It leverages

VisualGLM-6B 是一个多模态大语言模型和视觉语言系统，旨在处理并根据文本和视觉组合输入生成文本。它是一个双语对话 AI，能够维持英语和中文的自然语言交互。 该项目利用量化模型权重来降低内存需求，从而能够在消费级硬件上部署神经网络。这些压缩后的参数在降低显存占用的同时，保持了模型分析视觉内容并生成相应自然语言描述的能力。 该系统涵盖了包括视觉内容分析和自回归文本生成在内的一系列功能。它支持参数高效微调，以针对特定任务调整模型权重，并提供将模型作为网络服务托管以进行多模态对话的接口。

Open Flamingo 是一个多模态大语言模型训练框架，旨在将预训练的视觉编码器与语言模型集成。它实现了一种视觉-语言架构，利用交叉注意力层来处理交错的图像和文本序列。 该系统的特点是其少样本（few-shot）多模态学习能力，允许模型使用提示中提供的少量图像-文本示例来适应新的视觉任务。它支持上下文学习和多模态文本生成，用于视觉问答和字幕生成等任务。 该框架包括一个分布式模型训练器，采用数据并行和梯度检查点技术进行跨多个 GPU 的内存优化。它还提供用于分片多模态数据集加载、并行模型评估的实用程序，以及用于托管大规模模型进行推理的基础设施。