How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
MiniCPM is a collection of small language models designed for local, on-device deployment in resource-constrained environments. The project focuses on running dense Transformer models on consumer hardware, including GPUs, CPUs, and Apple Silicon, without requiring custom code forks. The project distinguishes itself through heavy optimization for edge hardware, utilizing quantized weight compression in GGUF and MLX formats to reduce memory overhead. It implements advanced inference techniques such as speculative sampling and radix-tree prefix caching to accelerate generation speed and throughp
llmware is a Python framework for AI agent orchestration and model management, designed to coordinate multi-model workflows and autonomous agents. It provides a unified model catalog and standardized interface to execute specialized language models for complex research, analysis, and structured data generation. The project distinguishes itself through its heavy emphasis on local execution and quantized inference, allowing models to run on private infrastructure using CPU, GPU, and NPU acceleration via runtimes like ONNX and OpenVino. It features a specialized ability to translate natural lang
xFasterTransformer is an exceptionally optimized solution for large language models (LLM) on the X86 platform, which is similar to FasterTransformer on the GPU platform. xFasterTransformer is able to operate in distributed mode across multiple sockets and nodes to support inference on larger…
⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡
The main features of intel/intel-extension-for-transformers are: Hardware Optimized Inference.
Projects with overlapping indexed features include: infrasys-ai/aiinfra. openbmb/minicpm — MiniCPM is a collection of small language models designed for local, on-device deployment in resource-constrained… llmware-ai/llmware — llmware is a Python framework for AI agent orchestration and model management, designed to coordinate multi-model… intel/xfastertransformer — xFasterTransformer is an exceptionally optimized solution for large language models (LLM) on the X86 platform, which… fminference/flexgen — FlexGen is an inference engine for large language models designed for high-throughput execution on single or multiple… abdelfattah-lab/nitro — Lightweight Python Wrapper for OpenVINO, enabling LLM inference on NPUs.
