StableLM is a pre-trained transformer-based large language model designed for natural language generation and zero-shot inference. It functions as a causal language model that predicts the next token in a sequence to produce human-like text for conversational and creative writing tasks. The model is built as a fine-tunable base, allowing the adaptation of pre-trained weights to specific tasks or styles through custom dataset training and weight regularization. It utilizes rotary positional embeddings and flash-attention to optimize memory usage and processing efficiency during deployment on G
FlashAttention is an attention mechanism optimization library and machine learning acceleration framework designed to increase training speed and reduce memory footprint for large-scale neural network models. It functions as a collection of low-level CUDA kernels that optimize memory-bound operations to improve hardware utilization on graphics processing units. The library distinguishes itself through an input-output-aware algorithm design that minimizes data movement between different levels of memory. By employing kernel fusion and tiled matrix multiplication, it combines sequential operati
3FS is a distributed file system and RDMA storage cluster designed for high-performance AI training and inference workloads. It functions as a strongly consistent storage layer that utilizes a disaggregated architecture to pool SSDs and memory resources across multiple nodes. The system provides specialized storage implementations including an AI training checkpoint store for parallel state preservation and a distributed key-value cache store for decoder layer vectors to optimize inference processing. It ensures data integrity through chain replication and apportioned query distribution. The
ICML 2025 Spotlight ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
The repository is the PyTorch implementation of HyperAttention paper:
The main features of insuhan/hyper-attn are: Attention Optimization.
Open-source alternatives to insuhan/hyper-attn include: stability-ai/stablelm — StableLM is a pre-trained transformer-based large language model designed for natural language generation and… dao-ailab/flash-attention — FlashAttention is an attention mechanism optimization library and machine learning acceleration framework designed to… deepseek-ai/3fs — 3FS is a distributed file system and RDMA storage cluster designed for high-performance AI training and inference… deepseek-ai/deepep — DeepEP is a distributed model accelerator and expert-parallel communication library designed to optimize the training… deepseek-ai/deepgemm — DeepGEMM is a suite of specialized GPU kernels and a just-in-time compiler designed for low-precision matrix… bytedance/shadowkv — [ICML 2025 Spotlight] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference.