apple/ml-stable-diffusion

Features

• Text-to-Image Generators - Provides a local generative AI pipeline for producing high-resolution images from text prompts on Apple Silicon.
• Image Generation - Produces high-quality images from text prompts using local hardware acceleration on Apple Silicon.
• Apple Hardware Acceleration - Offloads image generation inference to Apple's neural engine and hardware-optimized formats for high efficiency.
• Diffusion Models - Implements Stable Diffusion image generation optimized for Apple Silicon via Core ML.
• Guided Generation Layers - Implements a ControlNet integration layer to guide image generation using edge and depth maps.
• Composition-Controlled Generators - Guides and constrains visual output using ControlNet with edge and depth maps.
• Local AI Runtimes - Provides a complete local pipeline for text-to-image generation without cloud API dependencies.
• Model Conversion Pipelines - Transforms deep learning checkpoints into specialized Core ML formats for optimized execution on Apple hardware.
• Model Optimization Suites - Provides a suite of tools for quantizing weights and activations to reduce NPU latency.
• On-Device Inference Engines - Optimizes on-device inference speed and memory usage via weight and activation compression.
• Weight Quantization - Reduces model size using palettization and quantization to lower memory usage and improve mobile latency.
• AI Application Frameworks - Provides dedicated libraries for integrating generative image pipelines into native macOS and iOS applications.
• Sequential Model Splitting - Divides large generative models into sequential chunks to operate within Apple hardware memory constraints.
• Hardware Compute Planning - Estimates execution costs and identifies target hardware devices for specific model operations to predict performance.
• Inference Latency Optimizers - Configures specific hardware compute units and optimization hints to lower response times for model predictions.
• Hardware Memory Splitting - Divides large models into smaller chunks to fit within Apple Silicon hardware memory limits.
• Hardware Acceleration - Estimates execution costs and assigns operations to neural processing units for maximum performance.
• Model Format Converters - Converts deep learning checkpoints into Core ML formats for deployment on Mac and iOS.
• Model Loading Utilities - Loads pre-compiled model data and memory-mapped weights to decrease startup time for large generative models.
• Diffusion Process Conditioners - Implements mechanisms to constrain and direct the visual output of the diffusion process using additional input models.
• Activation Quantization - Lowers memory usage by estimating activation ranges from calibration data to apply precision reduction.
• Quantized Inference Runtimes - Implements low-latency inference by utilizing lower precision formats for weights and activations.
• Palettization - Reduces model memory footprint by mapping weights to a smaller set of representative values.
• Multilingual Encodings - Uses system-level textual embeddings and projection layers to support image generation from diverse languages and scripts.
• Dynamic Memory Allocation - Dynamically loads and unloads model resources during the generation process to reduce the system memory footprint.
• Model Safety Filters - Includes a dedicated safety model to verify that generated images adhere to content safety guidelines.
• Generative AI and LLM Tools - Optimized inference for diffusion models on Apple silicon hardware.

Star history