Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and
Chat with your favourite LLaMA models in a native macOS app
Enchanted is a privacy-focused, cross-platform chat frontend for interacting with self-hosted large language models on iOS and macOS. It serves as a native client for communicating with private model servers, specifically providing integration for the Ollama API. The application supports multimodal interactions, allowing users to combine text, image attachments, and voice prompts. It provides tools for local AI model management, including the ability to define persistent system prompts and switch between different models for specific tasks. The interface includes capabilities for rendering m
llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language models with hardware acceleration. It functions as a GGUF model loader and a structured text generator capable of running inference servers and multimodal runtimes for processing both text and image inputs. The project distinguishes itself through a local inference server that exposes model capabilities via an OpenAI-compatible web API. It supports advanced execution techniques including speculative decoding, weight quantization, and layer-based GPU offloading to manage memory acro
The main features of aphrodite-engine/aphrodite-engine are: Inference Engines.
Open-source alternatives to aphrodite-engine/aphrodite-engine include: ai-dynamo/dynamo — Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system… alexrozanski/llamachat — Chat with your favourite LLaMA models in a native macOS app. augustdev/enchanted — Enchanted is a privacy-focused, cross-platform chat frontend for interacting with self-hosted large language models on… b4rtaz/distributed-llama — Distributed-llama is a distributed inference engine and command line tool for running large language models across… bentoml/openllm — OpenLLM is a framework for deploying, managing, and scaling open-source large language models. abetlen/llama-cpp-python — llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language…