How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
The main features of josstorer/rwkv-runner are: Local Model Runners, Local LLM Tools, OpenAI-Compatible, Chat Interfaces, State-Based Inference Engines, Gated Linear Recurrent Layers, Model Launchers, OpenAI-Compatible Servers.
Projects with overlapping indexed features include: lostruins/koboldcpp — KoboldCPP is a local large language model inference engine and GGUF model runner designed to execute quantized models… ericlbuehler/mistral.rs — mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and… abetlen/llama-cpp-python — llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language… meta-pytorch/torchtune — Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a… coaidev/coai — CoAI is an enterprise-grade, self-hostable AI gateway platform that unifies access to over 200 AI models from more… antirez/ds4 — ds4 is a local inference engine for DeepSeek models that includes a distributed runtime for splitting transformer…
KoboldCPP is a local large language model inference engine and GGUF model runner designed to execute quantized models on personal hardware. It functions as a multimodal AI server and API gateway, providing OpenAI-compatible endpoints that allow third-party clients to interact with locally hosted models. The project distinguishes itself as an AI storytelling backend, featuring dedicated tools for long-form narrative management through persistent memory, world lore tracking, and character state management. It further extends its capabilities as a multimodal server capable of processing text, im
mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe
llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language models with hardware acceleration. It functions as a GGUF model loader and a structured text generator capable of running inference servers and multimodal runtimes for processing both text and image inputs. The project distinguishes itself through a local inference server that exposes model capabilities via an OpenAI-compatible web API. It supports advanced execution techniques including speculative decoding, weight quantization, and layer-based GPU offloading to manage memory acro
Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip