intel/ipex-llmArchived

Features

• Distributed Inference Engines - Provides a distributed inference engine to scale large models across multiple accelerators using pipeline and tensor parallelism.
• Weight Quantization - Converts model weights to INT4 or FP8 precision to reduce memory footprint and increase inference throughput.
• XPU Backend Integrations - Provides optimized execution kernels specifically for Intel GPUs and NPUs.
• OpenAI-Compatible APIs - Creates a compatible interface for serving models so external clients can use standard OpenAI-compatible web tools.
• Distributed Model Execution - Scales large model execution across multiple GPUs using tensor and pipeline parallelism.
• NPU Inference Execution - Executes quantized large language models on neural processing units via a portable command-line interface.
• Weight Conversions - Transforms model weights and configurations into formats compatible with Intel NPU execution.
• Local RAG Pipelines - Executes retrieval-augmented generation workflows using language and embedding models on local hardware.
• Model Inference Accelerators - Runs large language models on Intel GPUs and NPUs using quantization to increase speed.
• Model Inference Servers - Launches network API servers that follow industry standards to serve large language models to remote clients.
• Continuous Batching Strategies - Maximizes throughput by dynamically adding and removing sequences in active inference batches.
• GGUF Execution - Executes inference for large language models in GGUF format using optimized CPU and GPU backends.
• Model Quantization Tools - Includes tools for converting model weights to INT4, FP8, and GGUF formats to reduce memory and increase speed.
• Quantized Fine-Tuning - Optimizes model training on CPUs using 4-bit quantization to significantly reduce memory requirements.
• Runtime Precision Conversion - Transforms linear layers into low-bit integers during the model loading phase to accelerate execution.
• Multi-GPU Distribution - Allocates model computation across multiple GPUs to handle models exceeding single-device memory.
• Multimodal Models - Executes large multimodal models on local hardware to process combined text and image inputs.
• Multimodal Runtimes - Provides a specialized runtime for executing vision-language and speech-to-text models on local graphics processors.
• Parameter Efficient Fine-Tuning - Adapts pre-trained models using parameter-efficient techniques like QLoRA on Intel hardware.
• Parameter-Efficient Tuning Techniques - Updates a small subset of model weights using techniques like QLoRA to adapt models on limited hardware.
• Quantized Model Implementations - Imports and loads models using industry-standard quantization schemes such as GGUF, AWQ, and GPTQ.
• Quantized Model Runners - Provides a runtime environment designed to execute models quantized with AWQ using INT4 optimizations.
• Tensor Parallelism - Implements tensor parallelism to split model weights and computations across multiple accelerators.
• Text Embedding Generators - Executes embedding models to efficiently transform text into vector representations for semantic search.
• XPU Acceleration Toolkits - Serves as a comprehensive toolkit for accelerating LLM inference and finetuning on Intel CPUs, GPUs, and NPUs.
• KV-Cache Precision Scaling - Uses reduced precision for key-value caches to increase the total token capacity of the context window.
• Low-Bit Weight Quantization - Converts model weights to low-bit precision formats like INT4 and FP8 to maximize performance on Intel hardware.
• Local Agent Deployments - Runs large language models on local hardware to power autonomous agents with long-term memory.
• Visual Conversational State Management - Processes a sequence of text and image inputs to maintain a conversational context around visual elements.
• AI Orchestration Frameworks - Connects optimized model execution to orchestration tools to manage complex AI agent workflows.
• Audio Transcription - Processes audio input to generate text transcriptions using multimodal models.
• Speech-to-Text Translation - Converts recognized speech text from one language to another using generative models.
• Conversational AI Models - Executes conversational AI models to support real-time multi-turn dialogue and natural language interaction.
• Distributed Training Scaling Utilities - Provides utilities for managing and scaling model training workloads across distributed systems and multi-node clusters.
• Hardware Device Selection - Allows users to specify which individual GPUs to use for model execution when multiple accelerators are present.
• Inference Benchmarking Tools - Ships benchmark scripts to measure latency and throughput of model inference across Intel accelerators.
• RAG Pipelines - Implements workflows that ingest document data into vector databases for context-aware retrieval-augmented generation.
• Local RAG Implementations - Implements retrieval-augmented generation workflows utilizing local hardware for embeddings and generation.
• Long Context Processing - Processes and generates text using extended context windows on compatible graphics hardware.
• Mamba-specific Optimizations - Uses specialized APIs to reduce latency and memory usage specifically for Mamba-based models.
• Speculative Decoding Strategies - Uses draft models or self-speculation to predict multiple tokens in advance during text generation.
• Self-Speculative Decoding - Uses a low-precision draft predictor to validate sequences against a high-precision model to speed up generation.
• Prefill Phase Optimizations - Reduces memory usage during first token generation to support longer context windows.
• Mixture-of-Experts Inference Optimizers - Provides a specialized tool to accelerate the execution of mixture-of-experts models.
• LoRA Adapter Loaders - Loads multiple lightweight LoRA adapters on a base model to handle specialized requests efficiently.
• Multimodal Input Processors - Handles models that accept combined image, audio, or text inputs to generate text responses.
• Preference Optimization - Implements direct preference optimization to align model behavior with human preferences on Intel hardware.
• Prefix Caching - Implements prompt prefix caching to store KV caches and skip redundant computation for shared prefixes.
• Quantized Inference Runtimes - Supports running model inference using 4-bit integer quantization to reduce memory usage on compatible hardware.
• Text-to-Speech - Converts written text into spoken audio using optimized generative voice models.
• Multimodal Model Integrations - Executes vision and audio models on local hardware for visual question answering and transcription.
• Visual Question Answering - Predicts tokens based on combined image and text prompts to perform visual question answering.
• KV Cache Quantization - Uses FP8 precision for the key-value cache to increase the number of tokens stored in memory.
• PEFT Integrations - Enables the execution of parameter-efficient tuning workflows through integration with third-party PEFT libraries.
• VRAM & Compute Optimization - Implements techniques for balancing memory consumption and processing load to maximize hardware efficiency during inference.
• Knowledge Base Construction - Processes knowledge files to create searchable vector-based knowledge bases for question-answering tasks.
• Automated Workflow Integration - Connects locally hosted models to development platforms for automated processes like retrieval-augmented generation.
• Command Line Model Inferences - Provides a command-line interface for executing model inferences with configurable sampling parameters.
• Inference Runtime Integrations - Connects acceleration capabilities to external runtime environments to streamline the deployment of various model formats.
• Quantized Model Persistence - Saves and loads models using low-bit quantization to reduce memory footprint and avoid repeated optimization.
• RAG Pipeline Optimizers - Optimizes inference within retrieval-augmented generation engines to improve performance on Intel GPUs.
• VRAM Capacity Benchmarking - Allows benchmarking hardware to determine the maximum supported sequence length before running out of memory.
• Hardware Affinity Pinning - Provides the ability to pin worker processes to specific hardware sockets or cores to maximize throughput.
• Model Serving & Deployment - Runs LLMs on Intel hardware with low latency.

Star history