apple/ml-stable-diffusion
Ml Stable Diffusion
This project is a framework for running Stable Diffusion image generation models on Apple Silicon using Core ML hardware acceleration. It provides a local generative AI pipeline for producing images from text prompts using Swift and Python without relying on external cloud APIs.
The system includes a model converter to transform deep learning checkpoints into Core ML formats and a model optimizer to quantize weights and activations. It features a ControlNet integration layer to guide image generation using external signals such as edge and depth maps.
Capabilities cover text-to-image generation with multilingual text encoding and image safety verification. Performance is managed through weight compression, palettization, and model splitting to fit within hardware memory constraints, while compute planning and quantization are used to reduce prediction latency.
The implementation provides native interfaces for both Python and Swift to integrate generative pipelines into macOS and iOS applications.
Features
- Text-to-Image Generators - Provides a local generative AI pipeline for producing high-resolution images from text prompts on Apple Silicon.
- Image Generation - Produces high-quality images from text prompts using local hardware acceleration on Apple Silicon.
- Apple Hardware Acceleration - Offloads image generation inference to Apple's neural engine and hardware-optimized formats for high efficiency.
- Diffusion Models - Implements Stable Diffusion image generation optimized for Apple Silicon via Core ML.
- Guided Generation Layers - Implements a ControlNet integration layer to guide image generation using edge and depth maps.
- Composition-Controlled Generators - Guides and constrains visual output using ControlNet with edge and depth maps.
- Local AI Runtimes - Provides a complete local pipeline for text-to-image generation without cloud API dependencies.
- Model Conversion Pipelines - Transforms deep learning checkpoints into specialized Core ML formats for optimized execution on Apple hardware.
- Model Optimization Suites - Provides a suite of tools for quantizing weights and activations to reduce NPU latency.
- On-Device Inference Engines - Optimizes on-device inference speed and memory usage via weight and activation compression.
- Weight Quantization - Reduces model size using palettization and quantization to lower memory usage and improve mobile latency.
- AI Application Frameworks - Provides dedicated libraries for integrating generative image pipelines into native macOS and iOS applications.
- Sequential Model Splitting - Divides large generative models into sequential chunks to operate within Apple hardware memory constraints.
- Hardware Compute Planning - Estimates execution costs and identifies target hardware devices for specific model operations to predict performance.
- Inference Latency Optimizers - Configures specific hardware compute units and optimization hints to lower response times for model predictions.
- Hardware Memory Splitting - Divides large models into smaller chunks to fit within Apple Silicon hardware memory limits.
- Hardware Acceleration - Estimates execution costs and assigns operations to neural processing units for maximum performance.
- Model Format Converters - Converts deep learning checkpoints into Core ML formats for deployment on Mac and iOS.
- Model Loading Utilities - Loads pre-compiled model data and memory-mapped weights to decrease startup time for large generative models.
- Diffusion Process Conditioners - Implements mechanisms to constrain and direct the visual output of the diffusion process using additional input models.
- Activation Quantization - Lowers memory usage by estimating activation ranges from calibration data to apply precision reduction.
- Quantized Inference Runtimes - Implements low-latency inference by utilizing lower precision formats for weights and activations.
- Palettization - Reduces model memory footprint by mapping weights to a smaller set of representative values.
- Multilingual Encodings - Uses system-level textual embeddings and projection layers to support image generation from diverse languages and scripts.
- Dynamic Memory Allocation - Dynamically loads and unloads model resources during the generation process to reduce the system memory footprint.
- Model Safety Filters - Includes a dedicated safety model to verify that generated images adhere to content safety guidelines.
- Generative AI and LLM Tools - Optimized inference for diffusion models on Apple silicon hardware.
