SJTU-IPADS/PowerInfer

Features

• Hybrid Execution Engines - Provides a hybrid execution engine that splits model computation between CPU and GPU with automatic memory offloading.
• Neuron Activation Offloading - Distributes computation between GPU and CPU by predicting which neurons will activate based on input patterns.
• CPU-GPU Workload Balancing - Implements a scheduling system that distributes computation tasks between the CPU and GPU to maximize processing efficiency.
• Inference Acceleration - Increases token throughput and processing speed by optimizing memory access and reducing active parameters.
• Large Language Model Serving - Hosts and exposes large language models via an API, enabling high-parameter models to run on consumer hardware.
• CPU-GPU Hybrid Runtimes - Balances workloads by executing active parameters on the GPU and fallback computations on the CPU.
• Local Inference Engines - Implements a software framework for running large language models on local hardware using hybrid CPU and GPU execution.
• Model Inference Optimizations - Accelerates inference by reducing active parameters and optimizing memory access patterns for constrained hardware.
• Weight Offloading - Distributes network components between GPU and CPU memory to balance VRAM consumption and execution speed.
• Weight Quantization - Implements 4-bit weight quantization to compress model parameters and reduce the memory footprint on consumer hardware.
• Model Layer Offloading - Manages VRAM and offloads model layers between CPU and GPU to run oversized models on consumer hardware.
• Neuron-Activation Preloading - Optimizes memory access by preloading frequent neurons onto the GPU and computing rare ones on the CPU.
• Predictive Memory Preloading - Uses a small predictor model to preload frequent neurons into VRAM while keeping rare ones in system memory.
• Cross-Hardware Model Inference - Enables model inference across diverse platforms including Windows, AMD devices, and mobile hardware.
• Local Model Serving - Deploys large language models on personal hardware to provide a web API for external applications.
• Model Sparsification - Reduces active parameters during execution to increase token throughput and improve processing speed.
• Weight Conversion Utilities - Transforms standard model weights into specialized sparse structures optimized for fast local retrieval and inference.
• Model Serving APIs - Provides a web service interface that allows external clients to send batched generation requests to hosted models.
• Model Serving Frameworks - Ships a web service framework for hosting large language models with batch generation and API access.
• Weight Quantization Tools - Provides a utility for compressing model weights into 4-bit integers to reduce memory requirements.
• Cross-Platform Execution - Supports the execution of large language models across various operating systems and hardware architectures.
• VRAM Budget Controllers - Controls GPU memory usage through configurable limits and index resets to prevent system crashes during execution.
• VRAM Budget Management - Manages GPU memory usage via budget settings and index resets to prevent system crashes during execution.
• Inference Frameworks - Fast serving optimized for consumer-grade hardware.
• Local LLM Execution - High-speed inference engine for deploying models locally.
• Model Pruning and Sparsity - Enables fast serving on consumer-grade hardware.
• Model Serving & Deployment - Leverages activation locality for CPU/GPU LLM inference.

Star history