How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
OpenLLM is a framework for deploying, managing, and scaling open-source large language models
Chat with your favourite LLaMA models in a native macOS app
Enchanted is a privacy-focused, cross-platform chat frontend for interacting with self-hosted large language models on iOS and macOS. It serves as a native client for communicating with private model servers, specifically providing integration for the Ollama API. The application supports multimodal interactions, allowing users to combine text, image attachments, and voice prompts. It provides tools for local AI model management, including the ability to define persistent system prompts and switch between different models for specific tasks. The interface includes capabilities for rendering m
LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model providers. It provides a standardized API interface that abstracts vendor-specific schemas, allowing developers to interact with diverse models through a single, consistent format. By acting as a central traffic management layer, it enables organizations to route, secure, and govern model interactions across multiple deployments. The platform distinguishes itself through its policy-driven architecture, which uses configuration-based routing to manage traffic distribution, load balanc
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
The main features of microsoft/minference are: Attention Optimization, Inference and Serving, Inference Engines.
Open-source alternatives to microsoft/minference include: bigai-nlco/tokenswift. alexrozanski/llamachat — Chat with your favourite LLaMA models in a native macOS app. bentoml/openllm — OpenLLM is a framework for deploying, managing, and scaling open-source large language models. berriai/litellm — LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model… augustdev/enchanted — Enchanted is a privacy-focused, cross-platform chat frontend for interacting with self-hosted large language models on… bklieger-groq/g1.