Awesome GitHub RepositoriesPositional Embedding Techniques

This project is a comprehensive framework for the entire lifecycle of transformer-based language models, supporting everything from foundational pretraining to specialized deployment. It provides a modular toolkit for defining neural network architectures, managing data preparation pipelines, and executing training routines across various scales. The framework is designed to handle the full model development process, including supervised fine-tuning, behavioral alignment, and the integration of agentic capabilities. What distinguishes this framework is its focus on efficient training and adva

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

Qwen-7B is a pretrained causal language model designed for natural language generation, text processing, and complex reasoning tasks. It is available as an instruction-tuned model optimized for conversational interactions and a tool-use model capable of executing function calls and interacting with external APIs. The project provides a quantized version of the model to reduce GPU memory usage and supports the development of autonomous agents that can execute code and perform functions to complete complex goals. The system covers a wide range of capabilities including model fine-tuning throug

Qwen2.5-VL ist ein autoregressiver multimodaler Transformer, der darauf ausgelegt ist, verschachtelte Sequenzen von Text- und visuellen Token zu verarbeiten. Er integriert visuelle Merkmalseinbettungen in einen gemeinsamen Sprachmodellraum, um modalübergreifendes Denken durchzuführen und kohärente Antworten oder strukturierten Layout-Code zu generieren. Das Projekt zeichnet sich durch Vision-Language-Action-Mapping aus, das es ihm ermöglicht, visuelle Schnittstellen wahrzunehmen und diese Wahrnehmung in umsetzbare Befehle für die Bedienung digitaler Bildschirme und Roboterhardware zu übersetzen. Es verwendet eine Bildkodierung mit dynamischer Auflösung und eine zeitliche Video-Indizierung, um diverse Bildgrößen und visuelle Sequenzen langer Dauer zu handhaben. Das Modell deckt ein breites Spektrum an Fähigkeiten ab, einschließlich mehrsprachiger optischer Zeichenerkennung für die Dokumentendigitalisierung, räumlicher Verankerung zur Lokalisierung von Objekten über Begrenzungsrahmen und der Analyse von Langform-Videoinhalten. Es unterstützt zudem multimodales mathematisches Denken, um Probleme mithilfe von Diagrammen und Grafiken zu lösen, und erweitert sein Verständnis auf eine Kontextlänge von einer Million Token.

Qwen2.5-Coder is a code-centric large language model designed to generate, complete, and analyze source code. It serves as a polyglot programming model capable of producing functional code across hundreds of different programming languages. The model is optimized for reasoning over extensive software repositories, utilizing a context window that supports up to one million tokens. It also functions as an agentic coding framework, executing multi-step workflows and browser tasks through specialized function call formats. Its capabilities include large-scale codebase analysis, intelligent parti

This project is a manual reconstruction of the Llama 3 transformer architecture implemented as a PyTorch neural network. It serves as a reference for the internal mathematical structure and tensor flow of a transformer-based language model designed for next token prediction. The implementation focuses on building the model from scratch using basic matrix operations and tensor manipulations. It demonstrates the manual construction of core components, including rotary positional embeddings, multi-head self-attention, and root mean square normalization. The codebase covers the full inference pi

ChatGLM3 is an open-weights large language model designed for bilingual conversational interactions in English and Chinese. It functions as a tool-augmented system capable of calling external functions and executing internal code to resolve complex tasks. The model utilizes four-bit quantization to reduce memory requirements, enabling inference on consumer hardware and diverse processing units including GPUs and CPUs. It features an expanded context window for processing and summarizing long documents and includes a supervised fine-tuning pipeline for adapting the model to specialized domains

This project is a machine learning research automation system designed to manage the full research lifecycle, from idea discovery to final paper submission. It utilizes markdown-based skill templates to execute autonomous research tasks and manage iterative loops of deep review and experimentation. The system distinguishes itself through integrated capabilities for academic communication and integrity auditing. It can automate the generation of LaTeX papers, conference slide decks, and evidence-grounded peer review rebuttals. To ensure rigor, it employs cross-model review routing and adversar

Implementation of Denoising Diffusion Probabilistic Model in Pytorch

LeetCUDA is a collection of high-performance GPU kernel libraries focusing on memory optimization, activation functions, and attention mechanisms. It serves as a reference library for CUDA kernel implementations, ranging from basic element-wise operations to complex neural network components, and provides Python bindings to integrate these kernels into deep learning workflows. The project is distinguished by its focus on low-level hardware optimizations. This includes the use of tensor cores for half-precision matrix multiplication, asynchronous data pipelining with double buffering, and shar

TinyLlama is a compact 1.1B parameter language model pretrained on a dataset of 3 trillion tokens. It is an edge AI model designed for high-performance text generation on memory-constrained devices. The project provides a distributed pretraining framework for training small language models across multiple GPUs and nodes. It also includes a finetuning toolkit for full-parameter weight adjustments to adapt the base model for chat and specific tasks. The system supports distributed large language model training and on-device text generation. Its architectural components include rotary positiona

Yi is a bilingual language model and foundation model designed for natural language processing, reasoning, and reading comprehension in both English and Chinese. It is built as a transformer-based architecture capable of general purpose text generation and conversational tasks. The model is distinguished by its ability to function as a long context system, processing and analyzing extended input sequences up to 200k tokens. It also supports quantized versions that use low-bit precision to reduce memory footprints, enabling execution on consumer-grade hardware. The project covers a broad rang

EMO ist ein KI-Porträt-Animator und Audio-zu-Video-Diffusionsmodell, das entwickelt wurde, um ausdrucksstarke Talking-Head-Videos zu generieren. Es verwandelt ein einzelnes statisches Porträtbild und eine Audiospur in ein synchronisiertes Video einer sprechenden Person. Das System konzentriert sich auf die Synthese digitaler Menschen und erzeugt hochauflösende Gesichtsbewegungen und emotionale Signale. Es synchronisiert Lippenbewegungen und Gesichtsausdrücke mit gesprochenen Sprachaufnahmen, um realistische Porträt-Animationen zu erstellen. Das Framework nutzt einen Diffusionsprozess und einen Cross-Modal-Alignment-Mechanismus, um das Timing zwischen Audiosignalen und visuellen Landmarks sicherzustellen. Es verwendet referenzbasierte Bildkonditionierung, um die Identitätskonsistenz zu wahren, sowie eine zeitliche Konsistenzschicht, um flüssige Bewegungen zwischen den Frames zu gewährleisten.

Open Llama is an open source large language model and pre-trained transformer designed as a permissively licensed alternative to proprietary weights. It serves as a base model reproduction of the Llama architecture, providing a set of weights for a decoder-only transformer. The project provides a transparently trained model based on the RedPajama dataset, supporting unrestricted commercial and research use. It includes systems for serving pre-trained weights in various sizes. The project covers natural language processing research and performance benchmarking through text quality evaluation

InternLM is a large language model and a comprehensive suite of weights designed for text generation and complex reasoning. It functions as an inference engine for serving responses, a fine-tuning framework for adjusting model weights, and a platform for building autonomous AI agents. The system is capable of processing long-context input sequences up to one million tokens for document analysis. It employs chain-of-thought reasoning to solve knowledge-intensive tasks by generating intermediate logic steps before producing a final answer. The project covers model weight optimization through s

DeepSeek-LLM ist ein Large Language Model und kausales Sprachmodell für die natürliche Sprachgenerierung. Es fungiert als mehrsprachiges System, das in der Lage ist, das nächste Token in einer Sequenz vorherzusagen, um Textvervollständigung und konversationelle Generierung durchzuführen. Das Modell ist auf logisches Schlussfolgern spezialisiert, insbesondere als Code- und Mathe-LLM. Dies ermöglicht komplexe Problemlösungen, einschließlich der Generierung von ausführbarem Code und der Lösung mathematischer Gleichungen durch schrittweise Analyse. Die breiteren Fähigkeiten des Systems decken konversationelle KI ab, einschließlich der Generierung von Chat-Antworten und Textsequenzen in mehreren Sprachen. Der Funktionsumfang erstreckt sich auf automatisierte Codegenerierung und die Produktion kohärenter Texte für verschiedene Schreibaufgaben.

GLM-4 is an open weights large language model designed as a multimodal chat system. It functions as a reasoning-focused and multilingual model capable of processing and generating responses across text and visual data types. The model is distinguished by its function-calling capabilities, allowing it to interface with external tools and APIs to execute tasks and retrieve real-time information. It is optimized for complex logical reasoning, mathematical problem solving, and deep research involving long-form content generation. Broad capabilities include multilingual text generation, the creat

OpenNMT-py is a PyTorch neural machine translation framework used for training and deploying neural machine translation and large language models. It functions as a distributed model training system, an inference engine, and a toolkit for fine-tuning large language models. The framework distinguishes itself with a dedicated toolkit for adapting large language models through low-rank adaptation, quantization, and instruction tuning. It also includes a neural machine translation server that allows trained models to be hosted and exposed via REST API endpoints. The project covers a broad range

This project is a collection of educational resources and technical guides focused on the development and implementation of large language models. It provides a comprehensive curriculum covering transformer architectures, training methods, and deployment strategies. The materials provide detailed instructions for building autonomous agents using reasoning loops and tool integration, as well as guides for fine-tuning models through supervised learning and preference optimization. It also includes tutorials for constructing retrieval augmented generation pipelines and implementing transformer m

Liger-Kernel is a collection of pre-built fused Triton kernels and patching utilities designed to accelerate large language model training. It provides drop-in kernel replacements for common LLM operations such as RMSNorm, cross-entropy loss, and attention, enabling increased throughput and reduced memory usage while preserving bitwise-exact gradients. The project serves as a toolkit for composing custom model architectures from individual optimized kernels and for patching pre-existing models with minimal code changes. The project distinguishes itself through its ability to perform runtime m