1 Repo
Awesome GitHub RepositoriesNeuron-Activation Preloading
Memory optimizations that preload frequently activated neurons into VRAM to reduce data transfer latency.
Distinct from GPU Memory Optimizations: More specific than general GPU memory optimizations; it targets the predictive preloading of specific model parameters based on activation patterns.
Explore 1 awesome GitHub repository matching operating systems & systems programming · Neuron-Activation Preloading. Refine with filters or upvote what's useful.
Awesome Neuron-Activation Preloading GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- sjtu-ipads/powerinfer
SJTU-IPADS/PowerInfer
9,568Auf GitHub ansehenPowerInfer is an inference engine and serving framework designed to run large language models on local hardware. It combines a hybrid CPU-GPU offloader, a quantization tool, and a sparse model optimizer to enable the execution of high-parameter models on consumer-grade devices. The system distinguishes itself through neuron-activation-based offloading, using a predictor model to preload frequent neurons into VRAM while keeping rare neurons in system memory. This hybrid execution model balances workloads between the GPU and CPU based on input patterns to optimize memory access and increase tok
Optimizes memory access by preloading frequent neurons onto the GPU and computing rare ones on the CPU.
C++
