How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Distributed-llama is a distributed inference engine and command line tool for running large language models across multiple networked machines. It functions as a compute cluster manager that coordinates worker nodes to share the computational load of a single model.
The main features of b4rtaz/distributed-llama are: Distributed Inference Frameworks, Distributed Inference Engines, Distributed Model Execution, Tensor-Parallel Inference Distributions, Multi-Node, Large Language Model Deployments, Interactive Model Inference Sessions, Distributed Layer Synchronizers.
Projects with overlapping indexed features include: intel/ipex-llm — Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning… huggingface/text-generation-inference — Text Generation Inference is a production-ready engine designed for the deployment and serving of large language… opennmt/ctranslate2 — CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and… paddlepaddle/fastdeploy — FastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models.… intel-analytics/bigdl — BigDL is a PyTorch acceleration framework and distributed inference engine designed for large language models. It… zml/zml — zml is a machine learning model compiler and cross-platform inference engine that transforms model descriptions into…
Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP
Text Generation Inference is a production-ready engine designed for the deployment and serving of large language models. It functions as a containerized runtime environment that manages model execution, scales across distributed hardware, and provides high-performance inference capabilities for demanding production environments. The project distinguishes itself through advanced optimization techniques, including continuous batching to maximize hardware utilization and tensor parallelism to shard large models across multiple accelerator cards. It supports efficient inference through custom com
CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di
FastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models. It provides the infrastructure to launch model services that process combined image, video, and text inputs, exposing these capabilities through a standardized, OpenAI-compatible API for chat and text completions. The project distinguishes itself through advanced inference pipeline engineering and GPU optimization. It employs speculative decoding, tensor parallelism, and a disaggregated execution model that separates prefill and decode phases across different hardware resourc