datawhalechina/tiny-universe

Features

• Transformer Architecture Implementation - Recreates the full Transformer architecture from the original paper using PyTorch.
• Jupyter Notebook Collections - Delivers multiple independent AI subsystem implementations as self-contained Jupyter notebooks.
• Gated Activation Computations - Provides a from-scratch implementation of gated MLP activation computation for transformer models.
• Agentic Reasoning Frameworks - Provides a ReAct agent framework for planning multi-step tasks and executing external tool APIs.
• Agentic Reasoning Loops - Implements a reasoning-acting cycle where the model iteratively generates thoughts and selects tools.
• Autonomous Agent Frameworks - Implements a ReAct-style agent that plans tasks and executes external tool APIs.
• Decoder Architectures - Constructs a decoder-only transformer from scratch with stacked self-attention and feed-forward layers.
• Diffusion Model Training - Provides a minimal DDPM implementation translating mathematical formulas into training and sampling code.
• Text-to-Image Generators - Generates images from text descriptions using a Stable Diffusion XL model.
• Graph Retrieval Augmented Generation - Provides a retrieval-augmented generation system combining vector databases with knowledge graphs for associative retrieval.
• Hyperparameter Configurations - Configures model hyperparameters like vocabulary size, hidden dimensions, and layer counts.
• RAG Pipelines - Constructs a RAG system from the ground up performing document retrieval and knowledge-grounded text generation.
• LLM Architecture Explainers - Provides a detailed walkthrough of Qwen2 internals including GQA, RoPE, and attention masks.
• Normalization Layers - Implements layer normalization from scratch to stabilize training in deep neural networks.
• From-Scratch - Builds and trains transformer-based language models from scratch using only basic frameworks for education.
• Transformer Encoder Implementations - Builds a transformer encoder from scratch to produce contextualized sequence representations.
• Triangular Mask Implementations - Implements triangular causal masks from scratch for decoder-only transformer training.
• Multi-Head Attention Mechanisms - Implements multi-head attention with parallel heads and concatenated outputs from scratch.
• Grouped-Query Attention - Implements grouped-query attention from scratch to reduce KV-cache memory in transformer models.
• Rotary Positional Embeddings - Applies rotary position embeddings from scratch to encode token positions in transformer models.
• Residual Connection Implementations - Implements residual connections from scratch to enable training of deep transformer models.
• RMS Normalizations - Applies RMSNorm normalization from scratch to stabilize training in transformer models.
• Sequence Decoders - Implements a transformer decoder that generates output sequences using masked self-attention and cross-attention.
• Autoregressive Text Generation - Produces output sequences one token at a time by feeding previous outputs back into the decoder.
• Self-Attention Implementations - Implements self-attention from scratch where queries, keys, and values derive from the same input.
• Autonomous Task Agents - Creates autonomous agents that use reasoning and acting patterns to plan tasks and execute APIs.
• ReAct Deployments - Constructs a ReAct-style agent that reasons, selects tools, and integrates results into responses.
• Iterative Noise Removal - Generates new images by iteratively denoising random Gaussian noise using a trained diffusion model.
• Knowledge Graph Construction Tools - Extracts entities and relationships from unstructured documents using LLMs and stores them as graph structures.
• GraphRAG Integrations - Constructs a minimal GraphRAG pipeline integrating graph construction, retrieval, reasoning, and generation.
• Educational Implementations - Offers from-scratch implementations of transformer architectures and language model internals for learning.
• Transformer Implementations - Builds a complete Transformer model from the original paper specification using only PyTorch.
• Attention Scoring Functions - Computes relevance between queries and keys using scaled dot-product attention.
• Custom Dataset Evaluators - Accepts user-provided SFT-format datasets to run inference and scoring on custom tasks.
• F1 Metric Scorers - Computes F1 scores from tokenized text overlap between generated and reference answers.
• Grounded Answer Generation - Feeds retrieved document segments to an LLM to produce grounded answers with citations.
• Diffusion Model Evaluators - Evaluates generated image quality using Inception Score and Fréchet Inception Distance.
• Image-Text Consistency Checkers - Checks image-text consistency using a vision-language model to identify prompt mismatches.
• Reinforcement Learning Fine-Tuning - Implements reinforcement learning algorithms to fine-tune language models based on reward signals.
• Document Chunking & Embedding - Splits documents into fixed-length segments with configurable overlap for retrieval pipelines.
• Large Language Model Integration - Provides a unified interface for calling LLMs for text cleaning, extraction, or generation.
• Causal Masking - Applies lower-triangular masks so each token can only attend to itself and preceding tokens.
• Dual-Engine Evaluation Pipelines - Ships a two-stage inference-and-evaluation pipeline scoring outputs with F1, ROUGE, and accuracy.
• Accuracy Calculators - Calculates accuracy as the proportion of exact matches between predicted and correct answers.
• Text Embeddings - Converts document text into dense vector representations for similarity-based retrieval.
• Vector Embeddings - Converts text documents into vector representations using pluggable embedding models.
• Diffusion Models - Implements a denoising diffusion probabilistic model for image generation from pure noise.
• Hierarchical Community Clustering - Partitions knowledge graphs into nested communities and generates LLM summaries for each level.
• Community Summarizations - Generates textual summaries for each community by feeding nodes and edges to an LLM.
• Entity Resolution - Identifies and merges multiple references to the same real-world entity using LLM comparison.
• Hybrid Vector-Graph Databases - Combines a Neo4j knowledge graph with a vector database for associative and similarity-based retrieval.
• Vector Storage - Persists document vectors locally and retrieves relevant segments via cosine similarity.
• Evaluation Pipelines - Runs model inference on a dataset then scores outputs against reference answers using F1, ROUGE, and accuracy.
• LLM Evaluation - Provides quantitative evaluation tools measuring generative quality with F1, ROUGE, and Inception Score.
• AI and Machine Learning - Resources for understanding small-scale language model architectures.
• Development Guides - Educational resources for understanding small-scale models.
• Development Techniques - Educational resources for AI model development.

Star history