deepseek-ai/Engram
Engram
Engram es un sistema dinámico de recuperación de conocimiento y framework de aumento de memoria para modelos de lenguaje grandes (LLM). Funciona como una capa de búsqueda de memoria escalable y un componente de arquitectura dispersa diseñado para fusionar el conocimiento estático del modelo con estados externos dinámicos para mejorar la veracidad y reducir las alucinaciones.
El sistema utiliza recuperación de memoria condicional y direccionamiento de memoria diferenciable para mapear tokens de entrada a índices específicos dentro de un almacén de memoria asociativa a gran escala. Esto permite al modelo aumentar sus parámetros totales disponibles almacenando pesos en tablas de búsqueda externas y activando solo los segmentos de conocimiento relevantes para una entrada dada.
El framework cubre la optimización de dispersión del modelo y el aumento escalable, utilizando recuperación clave-valor y fusión dinámica de parámetros para mejorar el rendimiento en tareas especializadas sin requerir un reentrenamiento completo de la red.
Features
- Knowledge Retrieval Systems - Implements a system for accessing stored information by fusing static weights with scalable lookup mechanisms.
- Memory Storage and Retrieval Systems - Combines static memory with dynamic states to improve knowledge retrieval and manage data sparsity.
- Learned Memory Augmentations - Fuses static knowledge with dynamic states to improve retrieval and increase sparsity in large language models.
- Differentiable Memory Addressing - Uses differentiable memory addressing to learn optimal mappings between input tokens and memory indices.
- Parameter State Fusion - Fuses static weights with dynamic memory states to adapt the model's knowledge for each input.
- Dynamic Memory Integration - Expands the knowledge capacity of language models by integrating dynamic external memory states during inference.
- Model Sparsity - Reduces computational overhead by using conditional memory lookups to increase structural sparsity.
- External Memory Scaling - Increases total available parameters by storing weights in external lookup tables instead of active layers.
- Conditional Weight Retrieval - Implements a conditional memory lookup mechanism to retrieve specific weights based on input tokens.
- Sparse Model Architectures - Optimizes memory usage by implementing an architecture that activates only relevant knowledge segments.
- Associative Weight Retrieval - Provides a query-based lookup mechanism to retrieve specific weight vectors from an associative memory store.
- Embedding-to-Memory Mapping - Maps input embeddings to specific memory locations to retrieve relevant weights during the forward pass.
- Memory-Based Augmentation - Enhances performance on specialized tasks using external memory modules without requiring full network retraining.
- Embedding Lookup Layers - Provides a scalable lookup layer for conditional memory retrieval of external knowledge.
