# ztxz16/fastllm

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/ztxz16-fastllm).**

4,779 stars · 471 forks · C++ · Apache-2.0

## Links

- GitHub: https://github.com/ztxz16/fastllm
- awesome-repositories: https://awesome-repositories.com/repository/ztxz16-fastllm.md

## Description

fastllm是后端无依赖的高性能大模型推理库。同时支持张量并行推理稠密模型和混合模式推理MOE模型，任意10G以上显卡即可推理满血DeepSeek。双路9004/9005服务器+单显卡部署DeepSeek满血满精度原版模型，单并发20tps；INT4量化模型单并发30tps，多并发可达60+。

## Tags

### Part of an Awesome List

- [AI & Machine Learning](https://awesome-repositories.com/f/awesome-lists/ai/ai-machine-learning.md) — High-performance large model inference library
- [LLM Utilities](https://awesome-repositories.com/f/awesome-lists/devtools/llm-utilities.md) — C++ acceleration library for high-speed model inference.