awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
b4rtaz avatar

b4rtaz/distributed-llama

0
View on GitHub↗

Distributed Llama

Distributed-llama is a distributed inference engine and command line tool for running large language models across multiple networked machines. It functions as a compute cluster manager that coordinates worker nodes to share the computational load of a single model.

The system utilizes tensor parallelism to shard model weights across different hosts, allowing the execution of models that exceed the memory capacity of a single piece of hardware. It includes a dedicated format converter to transform standard model files into a compatible binary layout optimized for distributed loading.

The engine provides capabilities for multi-node model execution, worker node management, and text generation through a server interface or interactive chat sessions.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Distributed Inference Frameworks - A distributed inference framework that runs large language models across multiple networked machines using tensor parallelism.
  • Distributed Inference Engines - An engine for splitting and executing large language model inference workloads across multiple networked nodes.
  • Distributed Model Execution - Executes large language model workloads spread across multiple networked machines to increase processing speed.
  • Tensor-Parallel Inference Distributions - Splits model weights across multiple network nodes using tensor parallelism to handle models exceeding single-device memory.
  • Multi-Node - Groups independent machines into a virtual compute resource to enable multi-node tensor parallelism.
  • Large Language Model Deployments - Enables the deployment and execution of massive models that exceed the memory capacity of a single piece of hardware.
  • Interactive Model Inference Sessions - Provides a command-line interface and server for interactive chat sessions and batch text generation.
  • Distributed Layer Synchronizers - Coordinates the forward pass to ensure each model layer finishes processing across all nodes before the next begins.
  • Model Format Converters - Translates standard model files into specialized binary formats compatible with the distributed inference engine.
  • Coordinator-Worker Orchestration - Implements a coordinator-worker architecture to manage the distribution of inference tasks across a cluster of backend nodes.
  • Distributed Loading Layouts - Provides a dedicated converter to translate model weights into a binary layout optimized for distributed memory mapping.
  • Compute Cluster Orchestration - Coordinates the lifecycle and configuration of a cluster of nodes to share the computational load of a single model.
  • Command Line Model Inferences - Offers a command line interface for generating text responses from a local or networked model cluster.
  • Cluster Node Management - Acts as a backend coordinator for managing the membership and lifecycle of worker nodes across different hosts.
  • LLM Format Converters - Transforms large language models into specialized formats optimized for distributed inference on private hardware.
  • Worker Node Management - Initializes and configures backend worker nodes on specific hosts and ports to participate in the compute cluster.
  • Socket Networking - Uses low-level network sockets for the exchange of tensor data and synchronization signals between worker nodes.
  • Inference Engines - Distributed LLM inference connecting home devices into a tensor-parallel cluster.
  • LLM Development Utilities - Cluster-based acceleration for local LLM inference.
2,837 stars·210 forks·C++·mit·49 views

Star history

Star history chart for b4rtaz/distributed-llamaStar history chart for b4rtaz/distributed-llama

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does b4rtaz/distributed-llama do?

Distributed-llama is a distributed inference engine and command line tool for running large language models across multiple networked machines. It functions as a compute cluster manager that coordinates worker nodes to share the computational load of a single model.

What are the main features of b4rtaz/distributed-llama?

The main features of b4rtaz/distributed-llama are: Distributed Inference Frameworks, Distributed Inference Engines, Distributed Model Execution, Tensor-Parallel Inference Distributions, Multi-Node, Large Language Model Deployments, Interactive Model Inference Sessions, Distributed Layer Synchronizers.

Which projects share features with b4rtaz/distributed-llama?

Projects with overlapping indexed features include: intel/ipex-llm — Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning… huggingface/text-generation-inference — Text Generation Inference is a production-ready engine designed for the deployment and serving of large language… opennmt/ctranslate2 — CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and… paddlepaddle/fastdeploy — FastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models.… intel-analytics/bigdl — BigDL is a PyTorch acceleration framework and distributed inference engine designed for large language models. It… zml/zml — zml is a machine learning model compiler and cross-platform inference engine that transforms model descriptions into…

Projects sharing features with Distributed Llama

These projects share indexed features with Distributed Llama. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • intel/ipex-llmintel avatar

    intel/ipex-llm

    8,836View on GitHub↗

    Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

    Python
    View on GitHub↗8,836
  • huggingface/text-generation-inferencehuggingface avatar

    huggingface/text-generation-inference

    10,775View on GitHub↗

    Text Generation Inference is a production-ready engine designed for the deployment and serving of large language models. It functions as a containerized runtime environment that manages model execution, scales across distributed hardware, and provides high-performance inference capabilities for demanding production environments. The project distinguishes itself through advanced optimization techniques, including continuous batching to maximize hardware utilization and tensor parallelism to shard large models across multiple accelerator cards. It supports efficient inference through custom com

    Pythonbloomdeep-learningfalcon
    View on GitHub↗10,775
  • opennmt/ctranslate2OpenNMT avatar

    OpenNMT/CTranslate2

    4,736View on GitHub↗

    CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di

    C++avxavx2cpp
    View on GitHub↗4,736
  • paddlepaddle/fastdeployPaddlePaddle avatar

    PaddlePaddle/FastDeploy

    3,700View on GitHub↗

    FastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models. It provides the infrastructure to launch model services that process combined image, video, and text inputs, exposing these capabilities through a standardized, OpenAI-compatible API for chat and text completions. The project distinguishes itself through advanced inference pipeline engineering and GPU optimization. It employs speculative decoding, tensor parallelism, and a disaggregated execution model that separates prefill and decode phases across different hardware resourc

    Pythonernieernie-45ernie-45-vl
    View on GitHub↗3,700
Compare all 30 related projects→