awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
jundot avatar

jundot/omlx

0
View on GitHub↗
17,112 stars·1,452 forks·Python·Apache-2.0·40 viewsomlx.ai↗

Omlx

omlx is a local inference server designed to run large language models, vision models, and embedding models on Apple Silicon. It provides a private alternative to industry-standard AI endpoints by hosting a local API gateway that mirrors OpenAI and Anthropic specifications.

The system distinguishes itself through specialized hardware optimizations, including continuous batching for high throughput and a tiered caching system that offloads memory blocks to SSD. It also functions as a Model Context Protocol host, enabling the integration of local models with external tools, agents, and structured function calling.

The project includes a web-based management dashboard for configuring model weights and sampling parameters, alongside a native macOS menu bar controller for server health monitoring. Broad capabilities cover vision-language task execution, automated model lifecycle management via least-recently-used eviction, and API key authentication for securing local endpoints.

Features

  • Local Model Inference Servers - Hosts language, vision, embedding, and reranker models locally on Apple Silicon via a unified inference server.
  • Apple Silicon Inference - Implements a local inference server specifically optimized for executing large language models on Apple M-series hardware.
  • OpenAI-Compatible APIs - Provides HTTP endpoints that mirror OpenAI and Anthropic specifications for seamless integration with existing AI tools.
  • Model Context Protocol - Integrates local models with external tools and agents using the standardized Model Context Protocol.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI
  • Apple Silicon GPU Accelerators - Leverages Apple Silicon GPU and Neural Engine acceleration for high-performance local model inference.
  • Local Model Management - Provides a graphical interface for downloading weights and configuring model sampling parameters and lifecycles.
  • Local LLM API Servers - Runs local LLM, vision, and embedding models on Apple Silicon with a standard API for external integration.
  • Continuous Batching Strategies - Implements continuous batching to maximize hardware throughput by dynamically grouping simultaneous inference requests.
  • Local LLM Execution - Executes large language model inference locally on compatible hardware to process text generation.
  • Private AI Infrastructure - Provides a private infrastructure for hosting and managing machine learning models locally to ensure data sovereignty.
  • Agent Third-Party Integrations - Facilitates simplified one-click connections between local models and external coding agents or tools.
  • Tiered Storage Offloading - Implements a tiered caching system that offloads memory blocks to SSD to avoid recomputing data.
  • Tool Call Executions - Parses model outputs to execute function calls and produce structured JSON through external protocols.
  • Model Context Protocol Servers - Functions as a server that integrates models with external tools and agents using the Model Context Protocol.
  • Automated Memory Eviction - Automatically manages memory usage by evicting least-recently-used models and pinning high-priority assets.
  • Multi-Model Servers - Runs multiple model types, including text, vision, and embeddings, within a single server process.
  • Vision-Language Inference - Executes vision-language tasks, processing multi-image chats and encoded image strings via optimized models.
  • Tiered Caching - Implements a tiered caching system that overflows memory blocks to SSD to avoid recomputation across restarts.
  • Model Memory Managers - Manages device memory by automatically evicting least-recently-used models and pinning high-priority assets.
  • Model Management Dashboards - MLX-LM monitors real-time performance and manages weights from remote repositories using a centralized web interface.
  • Web Chat Interfaces - Includes a built-in web browser interface for chatting with models, managing history, and uploading images.
  • AI Tools - Local LLM inference server for Apple Silicon.
  • Inference Engines - Menu-bar managed inference server for Apple Silicon devices.

    • Star history

    Star history chart for jundot/omlxStar history chart for jundot/omlx

    How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

    Frequently asked questions

    What does jundot/omlx do?

    omlx is a local inference server designed to run large language models, vision models, and embedding models on Apple Silicon. It provides a private alternative to industry-standard AI endpoints by hosting a local API gateway that mirrors OpenAI and Anthropic specifications.

    What are the main features of jundot/omlx?

    The main features of jundot/omlx are: Local Model Inference Servers, Apple Silicon Inference, OpenAI-Compatible APIs, Model Context Protocol, Apple Silicon GPU Accelerators, Local Model Management, Local LLM API Servers, Continuous Batching Strategies.

    Which projects share features with jundot/omlx?

    Projects with overlapping indexed features include: sgl-project/sglang — Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It… getstream/vision-agents. ericlbuehler/mistral.rs — mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and… openvinotoolkit/openvino — OpenVINO is an AI inference engine and model serving platform designed to execute optimized deep learning models… ravenscroftj/turbopilot — Turbopilot is a local large language model inference server designed to provide private code completions. It functions… mervinpraison/praisonai — PraisonAI is an autonomous AI agent platform that coordinates multiple LLM-powered agents for research, planning, and…

    Projects sharing features with Omlx

    These projects share indexed features with Omlx. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
    • sgl-project/sglangsgl-project avatar

      sgl-project/sglang

      29,079View on GitHub↗

      Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

      Pythonattentionblackwellcuda
      View on GitHub↗29,079
    • getstream/vision-agentsGetStream avatar

      GetStream/Vision-Agents

      6,029View on GitHub↗
      Pythonagentic-aiagentsai
      View on GitHub↗6,029
    • ericlbuehler/mistral.rsEricLBuehler avatar

      EricLBuehler/mistral.rs

      6,597View on GitHub↗

      mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

      Rustllmrustuqff
      View on GitHub↗6,597
    • openvinotoolkit/openvinoopenvinotoolkit avatar

      openvinotoolkit/openvino

      10,414View on GitHub↗

      OpenVINO is an AI inference engine and model serving platform designed to execute optimized deep learning models across CPUs, GPUs, and NPUs through a unified API. It includes a model optimization toolkit for converting, quantizing, and compressing models from various frameworks, alongside a specialized generative AI runtime for large language models. The project distinguishes itself through a plugin-based hardware acceleration layer that maps neural network operations to vendor-specific drivers. It features advanced execution mechanisms such as continuous batching, speculative decoding, and

      C++aicomputer-visiondeep-learning
      View on GitHub↗10,414
    Compare all 30 related projects→