8 repository-uri
Awesome GitHub Repositoriesllama.cpp Backend Runners
Loads quantized GGUF models using the llama.cpp backend for efficient CPU and GPU inference.
Distinct from Model Serving: Distinct from Model Serving: specifically focuses on the llama.cpp backend for running quantized models, not general model serving infrastructure.
Explore 8 awesome GitHub repositories matching devops & infrastructure · llama.cpp Backend Runners. Refine with filters or upvote what's useful.
Awesome llama.cpp Backend Runners GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- cocktailpeanut/dalai
cocktailpeanut/dalai
12,920Vezi pe GitHubThe simplest way to run LLaMA on your local machine
Executes LLaMA models locally using a simple command-line interface.
CSSaillamallm - haifengl/smile
haifengl/smile
6,387Vezi pe GitHubSmile is a comprehensive JVM machine learning library and statistical computing toolkit. It provides a suite of algorithms for classification, regression, and clustering, implemented natively for Java, Scala, and Kotlin. The project also functions as a deep learning framework, a natural language processing library, and an inference engine for large language models. The library distinguishes itself through GPU acceleration via LibTorch bindings and support for the ONNX model interchange format. It includes specialized capabilities for large language model inference, featuring Byte-Pair Encodin
Generates text responses from LLaMA-3 models with support for chat and streaming API serving.
Java - strands-agents/sdk-python
strands-agents/sdk-python
6,176Vezi pe GitHubThis is an open-source Python SDK for building and orchestrating production-grade AI agents. It provides a unified framework for creating conversational agents that can use tools, maintain state, and coordinate across multiple language model providers including OpenAI, Anthropic, Google, Amazon Bedrock, and locally-hosted models. The SDK supports multi-agent orchestration through graphs, teams, and swarms, allowing several specialized agents to collaborate on complex tasks. Agents can be composed as callable tools that other agents invoke, and the framework includes policy handlers that inspe
Connects to Meta-hosted Llama API endpoints to run inference without managing your own infrastructure.
Python - serge-chat/serge
serge-chat/serge
5,725Vezi pe GitHubSerge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym
Uses llama.cpp as the core inference engine to run GGUF model files locally without external API dependencies.
Sveltealpacadockerfastapi - nsarrazin/serge
nsarrazin/serge
5,725Vezi pe GitHubA web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.
Uses llama.cpp as the core inference runtime for running GGUF-format models locally with CPU-optimized performance.
Svelte - mostlygeek/llama-swap
mostlygeek/llama-swap
4,786Vezi pe GitHubLlama-swap is a local inference orchestrator and API gateway for large language models. It functions as an OpenAI API proxy that manages the lifecycle of multiple local model servers, automatically starting and stopping them to swap models based on incoming request identifiers. The project distinguishes itself through dynamic model swapping and hardware optimization. It utilizes a specialized matrix-based concurrency control to define which models can run simultaneously and employs cost-based eviction to remove inactive servers from memory based on relative resource costs. The system provide
Automatically manages the lifecycle of llama.cpp servers to swap models based on incoming request identifiers.
Go - sakurallm/sakurallm
SakuraLLM/SakuraLLM
4,618Vezi pe GitHubSakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends
Loads quantized GGUF models using the llama.cpp backend for efficient CPU and GPU inference.
Python - shawwn/llama-dl
shawwn/llama-dl
4,126Vezi pe GitHubllama-dl este un instrument de linie de comandă conceput pentru descărcarea de mare viteză a ponderilor modelelor de limbaj mari (LLM) prin cereri HTTP. Funcționează ca un utilitar specializat pentru preluarea ponderilor de machine learning de mai mulți gigaocteți de pe servere externe. Instrumentul permite achiziția fișierelor modelelor Llama fără a fi nevoie de clienți externi de partajare a fișierelor sau protocoale torrent. Se concentrează pe transferul eficient al ponderilor masive ale modelelor pentru a pregăti mediile locale pentru implementarea modelelor de limbaj mari. Implementarea utilizează descărcarea segmentată, programarea concurentă a cererilor și maparea fișierelor bazată pe metadate pentru a gestiona achiziția datelor. Asigură integritatea datelor prin validarea fișierelor pe bază de checksum și utilizează streaming-ul de date pe bucăți (chunked) pentru a scrie segmentele de fișiere pe disc.
Functions as a specialized utility for downloading large-scale Llama model files without external clients.
Shell
