Awesome GitHub Repositoriesllama.cpp Backend Runners

The simplest way to run LLaMA on your local machine

Smile is a comprehensive JVM machine learning library and statistical computing toolkit. It provides a suite of algorithms for classification, regression, and clustering, implemented natively for Java, Scala, and Kotlin. The project also functions as a deep learning framework, a natural language processing library, and an inference engine for large language models. The library distinguishes itself through GPU acceleration via LibTorch bindings and support for the ONNX model interchange format. It includes specialized capabilities for large language model inference, featuring Byte-Pair Encodin

This is an open-source Python SDK for building and orchestrating production-grade AI agents. It provides a unified framework for creating conversational agents that can use tools, maintain state, and coordinate across multiple language model providers including OpenAI, Anthropic, Google, Amazon Bedrock, and locally-hosted models. The SDK supports multi-agent orchestration through graphs, teams, and swarms, allowing several specialized agents to collaborate on complex tasks. Agents can be composed as callable tools that other agents invoke, and the framework includes policy handlers that inspe

Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym

A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.

Llama-swap 是一个用于大语言模型的本地推理编排器和 API 网关。它作为一个 OpenAI API 代理，管理多个本地模型服务器的生命周期，根据传入的请求标识符自动启动和停止服务器以切换模型。 该项目通过动态模型切换和硬件优化脱颖而出。它利用专门的矩阵式并发控制来定义哪些模型可以同时运行，并采用基于成本的驱逐策略，根据相对资源成本从内存中移除不活跃的服务器。 系统提供全面的模型管理功能，包括标识符别名、请求过滤以及容器或虚拟机的生命周期命令执行。它还包含可观测性工具，如可视化模型测试平台、实时系统性能监控，以及 API 密钥验证和 TLS 加密等安全功能。 配置更新通过动态重载处理，监控文件系统变化，无需手动重启。

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

llama-dl 是一个命令行工具，专为通过 HTTP 请求高速检索大语言模型权重而设计。它作为一个专门的实用程序，用于从远程服务器获取多 GB 级的机器学习权重。 该工具无需外部文件共享客户端或 Torrent 协议即可获取 Llama 模型文件。它专注于高效传输海量模型权重，为大语言模型部署准备本地环境。 该实现使用分段下载、并发请求调度和基于元数据的文件映射来管理数据获取。它通过基于校验和的文件验证确保数据完整性，并采用分块数据流将文件片段写入磁盘。