ztxz16/fastllm
Fastllm
fastllm 是一套用于模型权重转换、混合专家 (MoE) 运行时和张量并行的专用软件组件。它提供了一个兼容 OpenAI 的 API 服务器，通过标准化的请求格式公开大语言模型功能。
该项目具有一个张量并行框架，可将计算工作负载拆分到多个 GPU 上以加速执行。它包含一个针对混合专家架构优化的专用运行时，以及一个将模型权重转换为低精度格式以减少内存使用并提高吞吐量的量化工具。
系统涵盖了分布式推理的高级工作流，包括设备映射内存管理、动态批处理和混合模式执行。它还提供了一个用于模型管理和部署配置的命令行界面和终端用户界面。
Features
- OpenAI-Compatible Model Servers - Exposes LLM capabilities through a web interface that implements the OpenAI API specification for easy integration.
- LLM Inference Servers - Provides a production-ready server for hosting large language models on local hardware without heavy frameworks.
- Deep Learning Inference Engines - Provides a high-performance engine for executing dense and Mixture-of-Experts models without heavy deep learning frameworks.
- Framework-Free Runtimes - Executes large language models using a custom C++ runtime that avoids reliance on heavy deep learning framework binaries.
- Distributed Inference Engines - Provides an engine for splitting and executing large model workloads across multiple processors.
- Model Quantization Tools - Provides a utility to reduce the precision of model weights, decreasing memory usage and accelerating inference.
- Mixture of Experts - Handles Mixture-of-Experts architectures by routing tokens to specific expert layers during the forward pass.
- MoE Inference Runtimes - Provides an execution environment specifically optimized for the inference of Mixture-of-Experts architectures like DeepSeek.
- Tensor-Parallel Inference Distributions - Implements tensor parallelism to split large model workloads across multiple processors for improved performance.
- Model Quantization Frameworks - Implements a workflow to convert model weights into lower precision formats to reduce memory usage.
- Weight Quantization - Compresses floating point weights into four-bit integers to reduce memory footprint and increase throughput.
- Tensor Parallelism - Splits large model tensors across multiple GPUs to distribute computational load and memory requirements.
- Tensor Parallelism Frameworks - Implements a framework for splitting neural network layers across multiple GPUs to accelerate execution.
- Model Memory Managers - Controls the allocation of model weights across hardware resources to run large models on limited VRAM.
- DeepSeek Model Configurations - Supports the deployment of full-precision or quantized DeepSeek models across single or multiple GPU setups.
- Text Generation Streaming - Implements a mechanism to deliver generated text incrementally using dynamic batching to reduce perceived latency.
- CLI Model Management - Provides a command-line interface for downloading models and exporting converted weights to local storage.
- Dynamic Inference Batching - Groups multiple incoming requests into single execution passes to maximize GPU utilization and reduce token latency.
- Deployment Management TUIs - Ships a terminal-based user interface for launching model deployments and managing configuration profiles.
- AI & Machine Learning - High-performance large model inference library
- LLM Utilities - C++ acceleration library for high-speed model inference.
