FMInference/FlexLLMGenArchived
FlexLLMGen
FlexLLMGen is an inference engine and runtime designed to run large language models on a single GPU by combining weight compression with tensor offloading. It reduces model weight memory usage by approximately 70% through 4-bit quantization, and stores model parameters, attention cache, and hidden states across GPU, CPU, and disk to fit models larger than available GPU memory.
The project distinguishes itself through a throughput-oriented batching approach that processes multiple generation requests together in large batches to maximize throughput on a single GPU. It also supports distributed scaling by combining offloading with pipeline parallelism across multiple machines, accelerating generation when aggregated GPU memory is insufficient. Integration with the HELM framework enables execution of language model benchmarks like MMLU using offloaded models on a single GPU.
The system provides a complete toolchain for model serving, including a model weight compressor, a tensor offloading framework, and a throughput-oriented server. It handles batch inference request processing, distributed GPU pipeline parallelism, and single-GPU large model execution through its memory offloading and weight compression capabilities.
Features
- Single-GPU Inference Runtimes - Runs large language models on a single GPU by offloading weights and cache to CPU and disk to fit models larger than available memory.
- Throughput-Oriented Servers - Processes multiple generation requests together in large batches to maximize throughput on a single GPU.
- Single-GPU Scaling - Runs large language models with limited GPU memory by offloading weights and attention cache to CPU and disk.
- High Throughput Inference - Processes multiple generation requests together in large batches to maximize throughput on throughput-oriented workloads.
- Single-GPU Execution Engines - Executes large language models with limited GPU memory by offloading weights and attention cache to CPU and disk.
- 4-Bit Compressors - Reduces LLM weight memory usage by approximately 70% through 4-bit compression with minimal accuracy loss.
- Inference Batching - Processes multiple generation requests together in large batches to maximize throughput on a single GPU.
- 4-Bit Quantization Tools - Reduces model weight memory by approximately 70% using 4-bit quantization with minimal accuracy loss.
- Memory Offloading Frameworks - Stores model parameters, attention cache, and hidden states across GPU, CPU, and disk to fit models larger than available GPU memory.
- Inference Engines - An engine for running large language models on a single GPU with weight compression and tensor offloading to CPU or disk.
- Pipeline Parallelism Partitioners - Distributes model layers across multiple GPUs to accelerate generation when aggregated GPU memory is insufficient.
- GPU Parallelism Partitioners - Combines offloading with pipeline parallelism across multiple machines to accelerate generation when aggregated GPU memory is insufficient.
- Distributed Offloading Systems - Combines offloading with pipeline parallelism across multiple machines to accelerate generation when aggregated GPU memory is insufficient.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
