Awesome GitHub RepositoriesPrefill Phase Optimizations

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

gpt-fast is a PyTorch transformer inference engine designed for text generation using a native tensor library implementation. It provides a runtime for executing large language models without the need for external C++ extensions. The project implements speculative decoding to accelerate generation by using a small draft model for token prediction and a larger model for verification. It further optimizes performance through a compiled prefill stage and a multi-GPU tensor parallelism library that shards linear layers across multiple graphics processing units. Memory efficiency is managed throu

DeepSeek-VL2 ist ein multimodales Large Language Model und Vision-Language-System, das darauf ausgelegt ist, visuelle Szenen zu analysieren und beschreibenden Text zu generieren. Es fungiert als Modell für visuelle Fragenbeantwortung (VQA) und visuelle Verankerung (Visual Grounding), das in der Lage ist, Informationen aus Dokumenten zu extrahieren und spezifische Objekte oder Regionen innerhalb von Bildern basierend auf textuellen Beschreibungen zu lokalisieren. Das Projekt nutzt eine Mixture-of-Experts-Architektur, um kombinierte Bild- und Texteingaben zu verarbeiten. Es ist für die Inferenz durch inkrementelles Prefilling optimiert, was den GPU-Speicherbedarf auf Hardware reduziert. Das Modell deckt multimodale Datenanalyse und visuelles Dokumentenverständnis ab, einschließlich der Interpretation von Diagrammen und Layouts. Es führt visuelle Inferenz und Verankerung durch, um textuelle Anfragen mit entsprechenden visuellen Inhalten abzugleichen.

FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference. It provides optimized implementations for attention operations (including flash attention, page attention, multi-head latent attention, and cascade attention) using paged key-value caches, fused kernel composition, and just-in-time compilation. The library also includes specialized kernels for mixture-of-experts layers, block-scaled low-precision quantization (FP8, FP4), and distributed collective communication. What distinguishes FlashInfer is its fused all-reduce communicat

FastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models. It provides the infrastructure to launch model services that process combined image, video, and text inputs, exposing these capabilities through a standardized, OpenAI-compatible API for chat and text completions. The project distinguishes itself through advanced inference pipeline engineering and GPU optimization. It employs speculative decoding, tensor parallelism, and a disaggregated execution model that separates prefill and decode phases across different hardware resourc

mini-sglang is a collection of tools for large language model inference, serving as an OpenAI-compatible inference server, a memory-efficient prefill engine, and a tensor parallelism runtime. It also functions as a local batch processing engine for offline benchmarking and ablation studies. The project focuses on acceleration and memory management through a KV cache manager that reuses precomputed caches for shared request prefixes. It handles large model workloads by distributing tasks across multiple GPUs and manages peak memory consumption by splitting long input sequences into smaller chu