Awesome GitHub RepositoriesNormalization Layers

TinyLlama is a compact 1.1B parameter language model pretrained on a dataset of 3 trillion tokens. It is an edge AI model designed for high-performance text generation on memory-constrained devices. The project provides a distributed pretraining framework for training small language models across multiple GPUs and nodes. It also includes a finetuning toolkit for full-parameter weight adjustments to adapt the base model for chat and specific tasks. The system supports distributed large language model training and on-device text generation. Its architectural components include rotary positiona

This project is a collection of PyTorch learning resources and educational guides designed to teach the construction and training of neural networks. It serves as a comprehensive deep learning tutorial covering various model architectures and practical implementation strategies. The resources provide specific guidance on implementing computer vision tasks, such as image classification and synthetic imagery generation, as well as reinforcement learning agents using value networks and experience replay. It also covers sequential data modeling through recurrent networks and generative modeling u

DeepSeek-LLM is a large language model and causal language model designed for natural language generation. It functions as a multi-lingual system capable of predicting the next token in a sequence to perform text completion and conversational generation. The model is specialized for logical reasoning, specifically as a code and math LLM. This enables it to perform complex problem solving, which includes generating executable code and solving mathematical equations through step-by-step analysis. The system's broader capabilities cover conversational AI, including the generation of chat comple

x-transformers هي مكتبة PyTorch ومجموعة أدوات بحثية لبناء معماريات المحولات (transformers). توفر إطار عمل معيارياً لتنفيذ أبحاث المحولات التجريبية، بما في ذلك مجموعة من آليات الانتباه المتقدمة، وأدوات نمذجة التسلسلات الطويلة، وإطار عمل لمحولات الرؤية. يتميز المشروع بتركيزه على المكونات عالية الأداء والموفرة للذاكرة، مثل Flash Attention مع النوى المبلطة وانتباه الاستعلام المتعدد. كما ينفذ طرقاً متخصصة لتوسيع نوافذ السياق، بما في ذلك تكرار التسلسل وتضمينات الموضع الدورانية. تغطي المكتبة مجموعة واسعة من القدرات المعمارية، بما في ذلك مخططات تطبيع متنوعة لتحقيق استقرار التدريب، وشبكات التغذية الأمامية المبوّبة، وطوبولوجيا الطبقات المخصصة مثل شبكات Macaron. تدعم كلاً من إنشاءات المشفر وفك التشفير، وتوفر أدوات لتوليد التسلسل التلقائي ومهام الرؤية واللغة مثل وصف الصور.

TransformerLens is a library for mechanistic interpretability research designed to reverse engineer the learned algorithms within large language models. It provides a standardized framework for wrapping diverse transformer architectures, allowing researchers to extract, manipulate, and analyze internal activations and weights through a consistent interface. The project distinguishes itself through a comprehensive system of activation hooks that can capture, patch, and ablate internal tensors during the forward pass. It includes specialized utilities for decomposing fused projections, material