deepseek-ai/Engram
Engram
Engram is a dynamic knowledge retrieval system and memory augmentation framework for large language models. It functions as a scalable memory lookup layer and sparse architecture component designed to fuse static model knowledge with dynamic external states to improve factuality and reduce hallucinations.
The system utilizes conditional memory retrieval and differentiable memory addressing to map input tokens to specific indices within a large-scale associative memory store. This allows the model to increase its total available parameters by storing weights in external lookup tables and activating only the relevant knowledge segments for a given input.
The framework covers model sparsity optimization and scalable augmentation, using key-value retrieval and dynamic parameter fusion to enhance performance on specialized tasks without requiring a full retraining of the network.
Features
- Knowledge Retrieval Systems - Implements a system for accessing stored information by fusing static weights with scalable lookup mechanisms.
- Memory Storage and Retrieval Systems - Combines static memory with dynamic states to improve knowledge retrieval and manage data sparsity.
- Learned Memory Augmentations - Fuses static knowledge with dynamic states to improve retrieval and increase sparsity in large language models.
- Differentiable Memory Addressing - Uses differentiable memory addressing to learn optimal mappings between input tokens and memory indices.
- Parameter State Fusion - Fuses static weights with dynamic memory states to adapt the model's knowledge for each input.
- Dynamic Memory Integration - Expands the knowledge capacity of language models by integrating dynamic external memory states during inference.
- Model Sparsity - Reduces computational overhead by using conditional memory lookups to increase structural sparsity.
- External Memory Scaling - Increases total available parameters by storing weights in external lookup tables instead of active layers.
- Conditional Weight Retrieval - Implements a conditional memory lookup mechanism to retrieve specific weights based on input tokens.
- Sparse Model Architectures - Optimizes memory usage by implementing an architecture that activates only relevant knowledge segments.
- Associative Weight Retrieval - Provides a query-based lookup mechanism to retrieve specific weight vectors from an associative memory store.
- Embedding-to-Memory Mapping - Maps input embeddings to specific memory locations to retrieve relevant weights during the forward pass.
- Memory-Based Augmentation - Enhances performance on specialized tasks using external memory modules without requiring full network retraining.
- Embedding Lookup Layers - Provides a scalable lookup layer for conditional memory retrieval of external knowledge.
