3 repository-uri
Awesome GitHub Repositoriesllama.cpp Backend Servers
Loads quantized GGUF models and runs them on GPU with the llama.cpp backend for efficient inference serving.
Distinct from Model Serving: Distinct from Model Serving: specifically focuses on serving models via the llama.cpp backend, not general web-based model serving.
Explore 3 awesome GitHub repositories matching web development · llama.cpp Backend Servers. Refine with filters or upvote what's useful.
Awesome llama.cpp Backend Servers GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- serge-chat/serge
serge-chat/serge
5,725Vezi pe GitHubSerge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym
Ships a browser-based chat interface that uses llama.cpp as the core inference engine for local model execution.
Sveltealpacadockerfastapi - ngxson/smolvlm-realtime-webcam
ngxson/smolvlm-realtime-webcam
5,560Vezi pe GitHubThis is a webcam-based client for a local llama.cpp server that enables real-time object detection and vision-language model inference directly from a browser. It captures frames from the user's webcam at configurable intervals and sends them to a locally running inference server for analysis, displaying both detection results and textual scene descriptions as they are produced. The application distinguishes itself by combining object detection with vision-language scene description in a single real-time interface, all processed through a local llama.cpp server for private, offline operation.
A web interface that communicates with a local llama.cpp inference server to process webcam frames through a vision-language model.
HTML - sakurallm/sakurallm
SakuraLLM/SakuraLLM
4,618Vezi pe GitHubSakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends
Loads quantized GGUF models and runs them on GPU with the llama.cpp backend for efficient inference serving.
Python
