Awesome GitHub Repositoriesllama.cpp Backend Runners

The simplest way to run LLaMA on your local machine

Smile is a comprehensive JVM machine learning library and statistical computing toolkit. It provides a suite of algorithms for classification, regression, and clustering, implemented natively for Java, Scala, and Kotlin. The project also functions as a deep learning framework, a natural language processing library, and an inference engine for large language models. The library distinguishes itself through GPU acceleration via LibTorch bindings and support for the ONNX model interchange format. It includes specialized capabilities for large language model inference, featuring Byte-Pair Encodin

This is an open-source Python SDK for building and orchestrating production-grade AI agents. It provides a unified framework for creating conversational agents that can use tools, maintain state, and coordinate across multiple language model providers including OpenAI, Anthropic, Google, Amazon Bedrock, and locally-hosted models. The SDK supports multi-agent orchestration through graphs, teams, and swarms, allowing several specialized agents to collaborate on complex tasks. Agents can be composed as callable tools that other agents invoke, and the framework includes policy handlers that inspe

Serge is a self-hosted web chat interface for running large language models locally using the llama.cpp inference engine. It loads GGUF-format model files directly on your own machine, removing the need for internet connectivity or external API keys, and streams responses to the browser in real time via WebSocket connections. The project is packaged for containerized deployment using Docker and Docker Compose, with a Traefik reverse proxy that handles HTTP and WebSocket routing along with automatic TLS certificate management. Ready-made Kubernetes manifests are also provided, enabling deploym

A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.

Llama-swap is a local inference orchestrator and API gateway for large language models. It functions as an OpenAI API proxy that manages the lifecycle of multiple local model servers, automatically starting and stopping them to swap models based on incoming request identifiers. The project distinguishes itself through dynamic model swapping and hardware optimization. It utilizes a specialized matrix-based concurrency control to define which models can run simultaneously and employs cost-based eviction to remove inactive servers from memory based on relative resource costs. The system provide

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

llama-dl هي أداة سطر أوامر مصممة للاسترجاع عالي السرعة لأوزان نماذج اللغات الكبيرة عبر طلبات HTTP. تعمل كأداة متخصصة لجلب أوزان تعلم الآلة التي تصل أحجامها إلى عدة جيجابايت من خوادم بعيدة. تتيح الأداة الحصول على ملفات نماذج Llama دون الحاجة إلى عملاء مشاركة ملفات خارجيين أو بروتوكولات تورنت. تركز على النقل الفعال لأوزان النماذج الضخمة لإعداد البيئات المحلية لنشر نماذج اللغات الكبيرة. يستخدم التنفيذ التنزيل المجزأ، وجدولة الطلبات المتزامنة، وتعيين الملفات المستند إلى البيانات الوصفية لإدارة الحصول على البيانات. تضمن سلامة البيانات من خلال التحقق من الملفات استناداً إلى المجموع الاختباري (checksum) وتستخدم بث البيانات المجزأ لكتابة أجزاء الملف على القرص.