gpt-fast is a PyTorch transformer inference engine designed for low-latency text generation. It functions as a distributed GPU inference library, a quantized model runner, and a speculative decoding framework. The system utilizes a speculative decoding workflow where a small draft model predicts token sequences for verification by a larger model to accelerate generation. It supports quantized model execution to reduce memory footprint and implements tensor parallelism to split computations across multiple GPUs. The project includes a standardized evaluation harness to measure the accuracy an
gpt-fast is a PyTorch transformer inference engine designed for text generation using a native tensor library implementation. It provides a runtime for executing large language models without the need for external C++ extensions. The project implements speculative decoding to accelerate generation by using a small draft model for token prediction and a larger model for verification. It further optimizes performance through a compiled prefill stage and a multi-GPU tensor parallelism library that shards linear layers across multiple graphics processing units. Memory efficiency is managed throu
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
ICML 2024 Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
The main features of infini-ai-lab/triforce are: Speculative Decoding.
Open-source alternatives to infini-ai-lab/triforce include: pytorch-labs/gpt-fast — gpt-fast is a PyTorch transformer inference engine designed for low-latency text generation. It functions as a… meta-pytorch/gpt-fast — gpt-fast is a PyTorch transformer inference engine designed for text generation using a native tensor library… hao-ai-lab/lookaheaddecoding — [ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. infini-ai-lab/magicdec — [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding. leezythu/focusllm — FocusLLM: Scaling LLM’s Context by Parallel Decoding. smart-lty/parallelspeculativedecoding — [ICLR 2025] PEARL: Parallel Speculative Decoding with Adaptive Draft Length.