ztxz16/fastllm

Features

• OpenAI-Compatible Model Servers - Exposes LLM capabilities through a web interface that implements the OpenAI API specification for easy integration.
• LLM Inference Servers - Provides a production-ready server for hosting large language models on local hardware without heavy frameworks.
• Deep Learning Inference Engines - Provides a high-performance engine for executing dense and Mixture-of-Experts models without heavy deep learning frameworks.
• Framework-Free Runtimes - Executes large language models using a custom C++ runtime that avoids reliance on heavy deep learning framework binaries.
• Distributed Inference Engines - Provides an engine for splitting and executing large model workloads across multiple processors.
• Model Quantization Tools - Provides a utility to reduce the precision of model weights, decreasing memory usage and accelerating inference.
• Mixture of Experts - Handles Mixture-of-Experts architectures by routing tokens to specific expert layers during the forward pass.
• MoE Inference Runtimes - Provides an execution environment specifically optimized for the inference of Mixture-of-Experts architectures like DeepSeek.
• Tensor-Parallel Inference Distributions - Implements tensor parallelism to split large model workloads across multiple processors for improved performance.
• Model Quantization Frameworks - Implements a workflow to convert model weights into lower precision formats to reduce memory usage.
• Weight Quantization - Compresses floating point weights into four-bit integers to reduce memory footprint and increase throughput.
• Tensor Parallelism - Splits large model tensors across multiple GPUs to distribute computational load and memory requirements.
• Tensor Parallelism Frameworks - Implements a framework for splitting neural network layers across multiple GPUs to accelerate execution.
• Model Memory Managers - Controls the allocation of model weights across hardware resources to run large models on limited VRAM.
• DeepSeek Model Configurations - Supports the deployment of full-precision or quantized DeepSeek models across single or multiple GPU setups.
• Text Generation Streaming - Implements a mechanism to deliver generated text incrementally using dynamic batching to reduce perceived latency.
• CLI Model Management - Provides a command-line interface for downloading models and exporting converted weights to local storage.
• Dynamic Inference Batching - Groups multiple incoming requests into single execution passes to maximize GPU utilization and reduce token latency.
• Deployment Management TUIs - Ships a terminal-based user interface for launching model deployments and managing configuration profiles.
• AI & Machine Learning - High-performance large model inference library
• LLM Utilities - C++ acceleration library for high-speed model inference.

Star history