How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
GLake: optimizing GPU memory management and IO transmission.
Dynamic Memory Management for Serving LLMs without PagedAttention
TensorRT-LLM is a platform and toolkit designed for compiling, optimizing, and serving transformer-based models on accelerated hardware. It functions as a framework that transforms machine learning models into efficient execution graphs, providing an engine to refine these models for specific hardware to maximize throughput and minimize latency during text generation. The project distinguishes itself through advanced execution strategies that manage the entire inference pipeline. It utilizes kernel-level fusion and static graph execution to optimize mathematical operations and computational f
Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training
SpotServe: Serving Generative Large Language Models on Preemptible Instances
The main features of hsword/spotserve are: Inference Serving Engines.
Open-source alternatives to hsword/spotserve include: intelligent-machine-learning/glake — GLake: optimizing GPU memory management and IO transmission. microsoft/vattention — Dynamic Memory Management for Serving LLMs without PagedAttention. nvidia/tensorrt-llm — TensorRT-LLM is a platform and toolkit designed for compiling, optimizing, and serving transformer-based models on… rulinshao/lightseq — Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training.