awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
Michael-A-Kuykendall avatar

Michael-A-Kuykendall/shimmy

0
View on GitHub↗

Shimmy

Shimmy is a local large language model inference engine and server that loads and serves GGUF formatted weights. It is distributed as a single binary runtime written in Rust, providing a standalone environment for running models without external runtime dependencies.

The project utilizes WebGPU for hardware acceleration, allowing model compute kernels to execute across diverse graphics hardware through a standardized interface. It features a local server that implements an OpenAI-compatible API layer, enabling applications to interface with local models via standardized REST endpoints.

Memory and performance are managed through quantized key-value cache compression to reduce GPU VRAM usage and rotary embedding scaling to extend the model context window. The system also includes automatic model file discovery to scan and register compatible weights from local storage.

The server is managed via a dedicated command-line interface for controlling operations and verifying model generation.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Local Inference - Provides a standalone binary runtime for executing GGUF models on local hardware.
  • OpenAI-Compatible APIs - Provides standard HTTP endpoints that translate OpenAI chat requests into internal inference calls.
  • OpenAI-Compatible Model Servers - Implements an OpenAI-compatible HTTP server for drop-in integration of local GGUF models.
  • GGUF Inference Engines - Serves as a standalone binary runtime for executing GGUF models with hardware acceleration.
  • GGUF Model Loaders - Directly loads and initializes GGUF-format model files for local inference.
  • GGUF Model Loaders - Provides a pipeline for discovering and loading GGUF-format models for local serving.
  • WebGPU Accelerations - Uses the WebGPU API to execute compute kernels for hardware acceleration across diverse GPUs.
  • LLM Compute Pipelines - Implements a Rust-based inference engine using WebGPU compute shaders for GGUF models.
  • Cross-Hardware Model Inference - Enables model execution across diverse graphics hardware through a standardized WebGPU interface.
  • KV Cache Optimizations - Optimizes LLM memory through KV cache compression and context window extension.
  • Position Embedding Scaling - Extends the model context window by scaling the frequency of positional embeddings.
  • Context Window Scaling - Increases the processable text limit by scaling rotational embeddings.
  • KV Cache Quantizers - Reduces VRAM usage by converting high-precision key-value cache tensors into low-precision integers.
  • KV Cache Quantization - Implements hardware-accelerated quantization of the key-value cache to reduce GPU VRAM usage.
  • Single-Binary Distributions - Distributes the entire server and inference logic as a single standalone executable binary.
  • AI & Machine Learning - Privacy-focused AI inference server with OpenAI compatibility.
  • Inference and Serving - Rust-based inference server with OpenAI compatibility.
  • Inference Engines - Rust-based inference server with OpenAI API compatibility.
  • Model Serving Engines - Rust-based inference server with OpenAI API compatibility.
  • Natural Language Processing - Rust-based inference server for NLP models with OpenAI API compatibility.
5,428 stars·513 forks·Rust·Apache-2.0·35 views

Star history

Star history chart for michael-a-kuykendall/shimmyStar history chart for michael-a-kuykendall/shimmy

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does michael-a-kuykendall/shimmy do?

Shimmy is a local large language model inference engine and server that loads and serves GGUF formatted weights. It is distributed as a single binary runtime written in Rust, providing a standalone environment for running models without external runtime dependencies.

What are the main features of michael-a-kuykendall/shimmy?

The main features of michael-a-kuykendall/shimmy are: Local Inference, OpenAI-Compatible APIs, OpenAI-Compatible Model Servers, GGUF Inference Engines, GGUF Model Loaders, WebGPU Accelerations, LLM Compute Pipelines, Cross-Hardware Model Inference.

Which projects share features with michael-a-kuykendall/shimmy?

Projects with overlapping indexed features include: ggerganov/llama.cpp — llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across… lm-sys/routellm — RouteLLM is a routing framework and traffic manager designed to direct prompts between high-capability and low-cost… opennmt/ctranslate2 — CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and… langchain-ai/langchain — LangChain is an orchestration framework designed for building, managing, and deploying applications powered by large… nvidia/tensorrt-llm — TensorRT-LLM is a platform and toolkit designed for compiling, optimizing, and serving transformer-based models on… lostruins/koboldcpp — KoboldCPP is a local large language model inference engine and GGUF model runner designed to execute quantized models…

Projects sharing features with Shimmy

These projects share indexed features with Shimmy. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • ggerganov/llama.cppggerganov avatar

    ggerganov/llama.cpp

    116,912View on GitHub↗

    llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across various hardware architectures. It provides the core components for local model execution, including a dedicated model quantizer for compressing weights into the GGUF format and a system for generating text embeddings for semantic search. The project distinguishes itself through specialized memory and execution optimizations, such as block-wise weight quantization to reduce memory footprints and memory-mapped model loading. It supports structured text generation by using formal

    C++
    View on GitHub↗116,912
  • lm-sys/routellmlm-sys avatar

    lm-sys/RouteLLM

    5,080View on GitHub↗

    RouteLLM is a routing framework and traffic manager designed to direct prompts between high-capability and low-cost large language models. It functions as an API gateway that mimics the OpenAI specification to route requests across different model providers. The system optimizes operational costs by splitting traffic between model tiers based on predicted win rates and prompt complexity. It includes a calibration tool to analyze sample queries and determine the optimal cost-quality tradeoff for traffic distribution. The framework provides a tool for measuring the accuracy and cost efficiency

    Python
    View on GitHub↗5,080
  • opennmt/ctranslate2OpenNMT avatar

    OpenNMT/CTranslate2

    4,319View on GitHub↗

    CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di

    C++avxavx2cpp
    View on GitHub↗4,319
  • langchain-ai/langchainlangchain-ai avatar

    langchain-ai/langchain

    139,458View on GitHub↗

    LangChain is an orchestration framework designed for building, managing, and deploying applications powered by large language models. It provides a unified integration layer that normalizes disparate model provider APIs into a consistent set of primitives, enabling developers to build complex, multi-step AI workflows that manage state, memory, and tool execution. The project distinguishes itself through a durable execution runtime that maintains persistent state across long-running processes by checkpointing progress to external storage. It models agent workflows as directed graphs, allowing

    Pythonagentsaiai-agents
    View on GitHub↗139,458
Compare all 30 related projects→