thu-pacman/chitu
Chitu
Chitu is a distributed serving platform and orchestrator for large language model inference. It functions as a compute manager designed to deploy and scale model workloads across diverse hardware architectures, including GPUs, CPUs, and heterogeneous hardware clusters.
The platform enables model deployment across a wide range of targets, including NVIDIA GPUs, regional chipsets, and legacy hardware. It manages the execution of models across these varying environments to increase available computing capacity and optimize resource utilization.
The system includes capabilities for distributed inference orchestration and heterogeneous hardware scaling, allowing models to run on configurations ranging from single devices to large production clusters. It also incorporates concurrent traffic management and request queueing to maintain stability during high-demand workloads.
Features
- LLM Serving Architectures - Provides a high-performance engineering architecture for deploying and serving large language models at scale across clusters.
- Cross-Hardware Model Inference - Enables the execution of large language models across diverse hardware, including NVIDIA GPUs, regional chipsets, and legacy systems.
- Distributed Model Execution - Distributes model inference workloads across multiple compute devices to increase processing speed and resource utilization.
- Heterogeneous Hardware Scaling - Increases computing capacity by running inference across mixed hardware clusters and non-NVIDIA chipsets.
- Inference Scaling - Scales model inference capacity across configurations ranging from single devices to large-scale production clusters.
- Inference Deployment Orchestrators - Orchestrates the deployment and scaling of large language models across heterogeneous hardware clusters.
- Heterogeneous Orchestrators - Coordinates the distribution of LLM workloads across varying CPU and GPU architectures to optimize resource utilization.
- Hardware-Agnostic Deployment - Provides a runtime that abstracts chip-specific instructions to allow models to run across GPUs, CPUs, and regional accelerators.
- Distributed Inference Orchestrators - Implements a system for distributing model weights and computation tasks across multiple devices and network nodes to scale processing.
- High-Throughput Model Serving - Employs architectures designed to handle large volumes of concurrent LLM inference requests with stable performance.
- Hardware Kernel Switching - Switches between hardware-specific kernels based on the target compute device to maintain optimal inference performance.
- Model Inference Deployment - Deploys large language models into production environments across diverse hardware including GPUs and CPUs.
- Traffic Management - Controls request throughput and connection concurrency to ensure stable operation in high-demand production environments.
- Model Tensor Mapping - Provides specialized memory mapping to allocate model tensors across diverse hardware architectures for optimized inference performance.
- Cluster Load Balancing - Balances computational loads across available cluster nodes to ensure maximum hardware utilization during model inference.
- Concurrent Request Limits - Manages simultaneous inference requests through a structured buffer to maintain system stability during traffic spikes.
- Inference and Serving - High-performance inference framework.
- Inference Engines - High-performance inference framework for large models.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
