serge-chat/serge
Serge
Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections.
The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deployment with persistent storage, resource limits, and optional ingress with TLS termination for production-oriented setups.
Configuration is managed through files rather than a runtime database, covering model loading and chat settings. The curated model list supports a range of open-source model families in GGUF format.
Features
- Web Chat Frontends - Ships a browser-based chat interface that uses llama.cpp as the core inference engine for local model execution.
- Local Model Chat Interfaces - Provides a web chat interface that runs large language models locally using GGUF files, no internet or API keys required.
- Offline Chat Clients - Provides a private, offline-capable AI chat system using open-source models and local hardware without internet or API keys.
- Self-Hosted Chat Interfaces - Runs large language models locally on your own hardware with a web chat interface, no external API keys or internet required.
- Self-Hosted Deployments - Deploys a containerized AI chat service using Docker Compose or Kubernetes with TLS and persistent storage.
- Self-Hosted LLM Frontends - Provides a self-hosted web UI for running and interacting with open-source language models without external API dependencies.
- GGUF Model Loaders - Loads GGUF-format model files from a curated list of supported open-source families for local inference.
- Application Deployments - Ships ready-made Kubernetes manifests for deploying the chat service with ingress, TLS, and persistent storage.
- llama.cpp Backend Runners - Uses llama.cpp as the core inference engine to run GGUF model files locally without external API dependencies.
- Web Chat Interfaces - Provides a browser-based chat interface that uses llama.cpp as the core inference engine for local model execution.
- Web Chat Interfaces - Provides a browser-based chat interface for interacting with locally hosted language models without external API keys.
- Model Streaming Over WebSocket - Streams model-generated text token-by-token over WebSocket connections for real-time interaction in the browser.
- Docker Container Deployments - Packages the entire application into Docker containers for consistent deployment across environments.
- Docker Compose Deployments - Provides a Docker Compose deployment that runs the chat interface behind a Traefik reverse proxy with automatic TLS.
- AI Application Deployments - Packages llama.cpp and a web interface as a containerized application for easy local AI deployment with Docker Compose.
- Kubernetes Deployments - Provides Kubernetes manifests for deploying the chat interface with persistent storage, resource limits, and optional ingress with TLS.
- AI Application Deployments - Provides ready-made Kubernetes manifests for deploying a local AI chat service with ingress, TLS, and persistent storage.
- AI Workload Deployments - Deploys a local language model service on Kubernetes with ingress, TLS termination, and WebSocket support.
- Traefik Integrations - Integrates Traefik reverse proxy for routing HTTP and WebSocket traffic with automatic TLS certificate management.
- Application Development - Self-hosted chat interface for running local models.
- Artificial Intelligence - Dockerized web interface for chatting with local models.
- Large Language Models - Dockerized web interface for interacting with local language models.
