How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
SpotServe: Serving Generative Large Language Models on Preemptible Instances
Dynamic Memory Management for Serving LLMs without PagedAttention
TensorRT-LLM is a platform and toolkit designed for compiling, optimizing, and serving transformer-based models on accelerated hardware. It functions as a framework that transforms machine learning models into efficient execution graphs, providing an engine to refine these models for specific hardware to maximize throughput and minimize latency during text generation. The project distinguishes itself through advanced execution strategies that manage the entire inference pipeline. It utilizes kernel-level fusion and static graph execution to optimize mathematical operations and computational f
Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training
GLake: optimizing GPU memory management and IO transmission.
The main features of intelligent-machine-learning/glake are: Inference Serving Engines.
Open-source alternatives to intelligent-machine-learning/glake include: hsword/spotserve — SpotServe: Serving Generative Large Language Models on Preemptible Instances. microsoft/vattention — Dynamic Memory Management for Serving LLMs without PagedAttention. nvidia/tensorrt-llm — TensorRT-LLM is a platform and toolkit designed for compiling, optimizing, and serving transformer-based models on… rulinshao/lightseq — Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training.