Awesome GitHub RepositoriesOpenAI-Compatible Servers

Jan is a local language model desktop application and AI assistant orchestrator. It provides a unified interface for interacting with both resident models and remote cloud AI providers. The project functions as a host for the Model Context Protocol, connecting AI models to external tools and data sources. It also operates as an OpenAI compatible API server, exposing local models through a standardized server endpoint for other applications to query. The system supports the creation of specialized AI personas with custom instructions and allows for the management of hybrid model environments,

Jan is a desktop application that functions as a local artificial intelligence model runtime and an open-standard API server. It enables the execution of large language models directly on local hardware, ensuring that data remains private and accessible offline while providing a unified interface for managing model weights and inference runtimes. The platform distinguishes itself by offering a modular inference backend that allows users to swap execution engines based on hardware compatibility and performance needs. It acts as a cross-platform orchestrator, providing the ability to switch bet

ds4 is a local inference engine for DeepSeek models that includes a distributed runtime for splitting transformer layers across networked computers. It functions as a reasoning controller with a local weight streamer and an API server that streams chat completions via industry standard endpoints. The system employs a memory management model that loads model experts from disk on demand to execute models that exceed available system RAM. It provides controls for reasoning effort and model behavior steering, allowing the modification of response characteristics through activation directions. Th

llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language models with hardware acceleration. It functions as a GGUF model loader and a structured text generator capable of running inference servers and multimodal runtimes for processing both text and image inputs. The project distinguishes itself through a local inference server that exposes model capabilities via an OpenAI-compatible web API. It supports advanced execution techniques including speculative decoding, weight quantization, and layer-based GPU offloading to manage memory acro

MiniCPM is a collection of small language models designed for local, on-device deployment in resource-constrained environments. The project focuses on running dense Transformer models on consumer hardware, including GPUs, CPUs, and Apple Silicon, without requiring custom code forks. The project distinguishes itself through heavy optimization for edge hardware, utilizing quantized weight compression in GGUF and MLX formats to reduce memory overhead. It implements advanced inference techniques such as speculative sampling and radix-tree prefix caching to accelerate generation speed and throughp

CoAI is an enterprise-grade, self-hostable AI gateway platform that unifies access to over 200 AI models from more than 35 providers through a single OpenAI-compatible API endpoint. It functions as a multi-tenant gateway, routing requests across providers with load balancing, automatic failover, and priority-based routing, while exposing standard OpenAI API endpoints for chat, image generation, model listing, and billing to enable seamless integration with existing tools and clients. The platform distinguishes itself through a comprehensive set of operational capabilities built around the gat

Guardrails is a Python SDK that wraps calls to large language models with configurable validation pipelines, corrective actions, and structured output generation. It provides a unified API layer that connects to over 100 language models, applying consistent validation, streaming, and error-handling across providers. The framework validates and corrects model responses against safety and quality rules, detecting and mitigating risks in both inputs and outputs using pre-built and custom validators. The project distinguishes itself through a validator-pipeline architecture that sequentially appl

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

Superdesign is an AI-powered design platform that generates UI mockups, wireframes, and multi-page user flows from natural language prompts within a collaborative canvas environment. It functions as a design-to-code exporter, producing production-ready HTML, ZIP archives, or Shopify Liquid templates for direct implementation, and includes an OpenAPI specification importer that automatically generates API documentation and client code from schema definitions. The platform distinguishes itself through a branch-based design exploration system that creates independent design variations from a sin

The free AI already on your Mac. CLI tool, OpenAI-compatible server, and interactive chat — all on-device via Apple Intelligence. No API keys, no cloud, no downloads.

LiteRT-LM est un framework d'inférence haute performance conçu pour exécuter des grands modèles de langage localement sur du matériel mobile, desktop et IoT. Il sert de runtime de modèle sur appareil qui utilise l'accélération CPU, GPU et NPU pour fournir un traitement à faible latence. Le framework se distingue par sa capacité à traiter des entrées texte, vision et audio via un moteur d'inférence multi-modal unique. Il dispose d'un serveur HTTP local qui émule des endpoints d'API compatibles avec OpenAI et un runtime basé sur WebGPU pour exécuter des modèles directement dans un navigateur web. Pour garantir la fiabilité des sorties, il inclut un générateur de texte contraint qui impose des schémas JSON ou des règles grammaticales sur les réponses du modèle. Le projet offre de larges capacités pour la gestion de conversations stateful, le décodage spéculatif pour augmenter les vitesses de génération de jetons, et une interface d'appel d'outils qui mappe les requêtes du modèle vers des fonctions externes. Il inclut également une intégration spécialisée pour l'écosystème Apple et un plugin dédié pour exécuter des modèles dans Flutter. Les utilisateurs peuvent exécuter des modèles via une interface en ligne de commande ou les intégrer dans des applications via des API natives.

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere