intel-analytics/BigDLArchived
BigDL
BigDL is a PyTorch acceleration framework and distributed inference engine designed for large language models. It provides a toolkit for running models on Intel hardware, integrating quantization tools and libraries for parameter-efficient fine-tuning.
The project distinguishes itself through the use of pipeline parallelism to distribute model workloads across multiple hardware accelerators. It utilizes low-bit integer quantization and speculative decoding to reduce memory footprints and decrease text generation latency.
The system covers broad capabilities in model optimization, including weight compression and quantized model loading. It also supports hardware-accelerated training routines to adapt pre-trained models to specific tasks.
Features
- XPU Acceleration Toolkits - Optimizes compute kernels specifically for Intel CPUs and GPUs to improve inference and fine-tuning performance.
- PyTorch-Based Frameworks - Provides a toolkit for optimizing and executing PyTorch models on hardware accelerators via weight compression and parallelism.
- Distributed Inference Engines - Provides a distributed engine that splits large model workloads across multiple accelerators using pipeline parallelism.
- Distributed Inference Scaling - Scales inference by executing large scale models across multiple hardware accelerators via pipeline parallelism.
- Distributed Model Execution - Executes large model workloads across multiple compute devices to balance heavy computational loads.
- Hardware-Accelerated Inference - Optimizes model execution across different hardware processors to increase speed and reduce latency.
- Intel XPU LLM Inference - Runs large language models on Intel hardware using INT4 quantization for high-performance, low-latency inference.
- Large Language Model Fine-Tuning - Provides hardware-accelerated training routines and parameter-efficient tuning to adapt pre-trained models to specific tasks.
- Intel XPU - Ships a library for running large language models on Intel hardware using INT4 quantization.
- Pipeline Parallelisms - Distributes model layers across multiple hardware accelerators using pipeline parallelism to handle massive models.
- Parameter Efficient Fine-Tuning - Provides hardware-accelerated routines for adapting pre-trained models using parameter-efficient fine-tuning.
- PyTorch Backends - Interfaces with PyTorch to enable seamless loading and execution of standard model architectures on accelerated hardware.
- Weight Quantization - Implements weight quantization to compress model weights into low-bit formats, reducing memory footprint and increasing speed.
- Low-Bit Weight Quantization - Compresses LLM weights into low-bit precision formats to reduce memory usage and increase execution speed.
- Speculative Decoding Strategies - Decreases text generation latency by predicting and validating multiple tokens in a single forward pass.
- Self-Speculative Decoding - Implements self-speculative decoding to speed up text generation by predicting multiple tokens in parallel.
- Quantized Model Loading - Provides the ability to import models from common compressed formats for higher efficiency and lower resource overhead.
- Parameter-Efficient Training Toolkits - Implements a framework for adapting pre-trained models to specific tasks using hardware-accelerated, parameter-efficient tuning.
- PyTorch Model Optimizations - Accelerates the execution of PyTorch based language models by optimizing them for Intel XPU hardware targets.
- LLM Quantization Frameworks - Provides a system for reducing model memory usage by converting weights into low-bit formats.
- Large Language Models - Distributed deep learning library for big data platforms.
- Machine Learning - Distributed deep learning library.
- Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
