LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a multi-GPU inference engine and server capable of handling dense architectures, mixture-of-experts designs, and multimodal models that process both text and images. The system is distinguished by its specialized support for Mixture-of-Experts models using expert parallelism and fused kernels. It implements structured text generation through deterministic state machines and pushdown automata to enforce precise output formats. To optimize throughput, the framework employs specula
MXNet is a deep learning framework and distributed machine learning engine designed for training and deploying neural networks. It functions as a hardware-agnostic backend that allows for the development of deep learning models through a hybrid of symbolic and imperative programming. The system distinguishes itself through automatic distributed parallelism, which scales training workloads across multiple GPUs and machines. It features an extensible hardware backend interface that enables the integration of custom accelerators and proprietary libraries without modifying the core source code.
StableLM is a pre-trained transformer-based large language model designed for natural language generation and zero-shot inference. It functions as a causal language model that predicts the next token in a sequence to produce human-like text for conversational and creative writing tasks. The model is built as a fine-tunable base, allowing the adaptation of pre-trained weights to specific tasks or styles through custom dataset training and weight regularization. It utilizes rotary positional embeddings and flash-attention to optimize memory usage and processing efficiency during deployment on G
ICML 2025 Spotlight ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
Large Context Attention
The main features of lhao499/ringattention are: Attention Optimization, Distributed Parallelism.
Open-source alternatives to lhao499/ringattention include: dmlc/mxnet — MXNet is a deep learning framework and distributed machine learning engine designed for training and deploying neural… modeltc/lightllm — LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a… stability-ai/stablelm — StableLM is a pre-trained transformer-based large language model designed for natural language generation and… bytedance/shadowkv — [ICML 2025 Spotlight] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference. deepseek-ai/deepep — DeepEP is a distributed model accelerator and expert-parallel communication library designed to optimize the training… deepseek-ai/3fs — 3FS is a distributed file system and RDMA storage cluster designed for high-performance AI training and inference…