How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
MiniCPM is a collection of small language models designed for local, on-device deployment in resource-constrained environments. The project focuses on running dense Transformer models on consumer hardware, including GPUs, CPUs, and Apple Silicon, without requiring custom code forks. The project distinguishes itself through heavy optimization for edge hardware, utilizing quantized weight compression in GGUF and MLX formats to reduce memory overhead. It implements advanced inference techniques such as speculative sampling and radix-tree prefix caching to accelerate generation speed and throughp
llmware is a Python framework for AI agent orchestration and model management, designed to coordinate multi-model workflows and autonomous agents. It provides a unified model catalog and standardized interface to execute specialized language models for complex research, analysis, and structured data generation. The project distinguishes itself through its heavy emphasis on local execution and quantized inference, allowing models to run on private infrastructure using CPU, GPU, and NPU acceleration via runtimes like ONNX and OpenVino. It features a specialized ability to translate natural lang
xFasterTransformer is an exceptionally optimized solution for large language models (LLM) on the X86 platform, which is similar to FasterTransformer on the GPU platform. xFasterTransformer is able to operate in distributed mode across multiple sockets and nodes to support inference on larger…
Lightweight Python Wrapper for OpenVINO, enabling LLM inference on NPUs
The main features of abdelfattah-lab/nitro are: Hardware Optimized Inference.
Open-source alternatives to abdelfattah-lab/nitro include: infrasys-ai/aiinfra. openbmb/minicpm — MiniCPM is a collection of small language models designed for local, on-device deployment in resource-constrained… llmware-ai/llmware — llmware is a Python framework for AI agent orchestration and model management, designed to coordinate multi-model… intel/xfastertransformer — xFasterTransformer is an exceptionally optimized solution for large language models (LLM) on the X86 platform, which… intel/intel-extension-for-transformers — ⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs… fminference/flexgen — FlexGen is an inference engine for large language models designed for high-throughput execution on single or multiple…
