2 repositorios
Awesome GitHub RepositoriesSparse Inference Frameworks
Execution systems that optimize inference by skipping computations based on activation sparsity in model weights.
Distinct from Sparse Model Architectures: Distinct from Sparse Model Architectures: focuses on the execution runtime and skipping computations rather than the neural network design.
Explore 2 awesome GitHub repositories matching artificial intelligence & ml · Sparse Inference Frameworks. Refine with filters or upvote what's useful.
Awesome Sparse Inference Frameworks GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- tiiny-ai/powerinfer
Tiiny-AI/PowerInfer
8,714Ver en GitHubPowerInfer is a high-performance local large language model inference engine and sparse inference framework. It provides a runtime for executing models on consumer-grade hardware, utilizing a GPU acceleration backend to optimize tensor operations for graphics processors. The system distinguishes itself through a sparse inference framework that increases generation speed by skipping computations based on activation sparsity in model weights. It includes a GGUF model converter for transforming weights and metadata into a unified binary format, as well as an OpenAI API compatible server for inte
Implements a sparse inference framework that increases generation speed by exploiting activation sparsity.
C++large-language-modelsllamallm - paddlepaddle/fastdeploy
PaddlePaddle/FastDeploy
3,700Ver en GitHubFastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models. It provides the infrastructure to launch model services that process combined image, video, and text inputs, exposing these capabilities through a standardized, OpenAI-compatible API for chat and text completions. The project distinguishes itself through advanced inference pipeline engineering and GPU optimization. It employs speculative decoding, tensor parallelism, and a disaggregated execution model that separates prefill and decode phases across different hardware resourc
Uses sparse attention mechanisms to process key-value blocks selectively and handle long-sequence inputs.
Pythonernieernie-45ernie-45-vl
