vladmandic/sdnext

Features

• AI-Powered Image and Video Generation - Ships a comprehensive web interface for generating and editing images and videos using diffusion models.
• Hardware Abstraction Layers - Provides a hardware abstraction layer that auto-detects and configures CUDA, ROCm, DirectML, and OpenVINO.
• Adaptive Network Weights - Applies network modules like LoRA to steer the generation toward specific styles or subjects.
• Generative API Exposures - Exposes server capabilities to external tools and scripts through a programmable API.
• AMD Hardware Acceleration - Enables AMD GPU acceleration by leveraging ZLUDA and the ROCm stack.
• ControlNet Guidance - Uses ControlNet to steer image generation by constraining pose, depth, and style.
• CUDA Accelerated Neural Networks - Accelerates image generation through optimized neural network implementations on NVIDIA GPUs.
• Diffusion Model Managers - Provides tools for downloading, quantizing, and optimizing diffusion model weights to reduce VRAM usage.
• Diffusion Models - Provides a specialized web user interface for initializing and running diffusion-based image and video synthesis.
• Hardware Acceleration Backends - Provides a configuration layer to select specific GPU or accelerator backends for optimized model inference.
• Hardware Device Selection - Provides utilities to specify and target particular hardware accelerators for model execution and inference pipelines.
• Automated Processing Pipelines - Runs automated image pipelines for bulk operations such as captioning, tagging, and filtering.
• Image-Conditioned Generation - Generates or edits images using other images as starting points or visual references.
• Multi-Backend GPU Inference Engines - Provides a multi-backend inference engine that automatically detects and accelerates across CUDA, ROCm, OpenVINO, and DirectML.
• Multi-Backend Inference Support - Implements a multi-backend execution layer that runs AI models across diverse GPU and CPU accelerators.
• Model Lifecycle Managers - Provides a system for validating, converting, and managing the local storage of diffusion model weights.
• VRAM Offloading - Implements techniques to reduce GPU memory usage by offloading model components to system RAM during inference.
• Batch Image Processors - Provides batch image processing to apply generation or editing operations to multiple files simultaneously.
• Batch Processing Pipelines - Implements batch processing pipelines to automate sequential image captioning, upscaling, and filtering.
• Cross-Platform Execution - Executes generation and processing tasks across diverse GPU and CPU architectures with optimized runtimes.
• Headless Execution Modes - Provides a headless execution mode for remote access and scripted generation tasks.
• Multi-Backend Inference Executions - Utilizes the OpenVINO execution provider to accelerate model inference on Intel hardware.
• Headless APIs - Offers a headless REST API to trigger image and video generation without a graphical interface.
• Image Processing Pipelines - Implements sequential workflows that chain captioning, upscaling, and filtering for automated image processing.
• Inference Performance Optimizers - Optimizes inference performance by employing weight quantization and model memory offloading.
• Language Model Prompt Rewriters - Employs language models to rewrite and enhance simple text prompts for better visual output.
• Attention Kernel Optimizers - Utilizes optimized attention kernels to increase processing speed for diffusion model operations.
• Attention Slicing Tuners - Adjusts dynamic attention slicing thresholds to optimize model execution within specific GPU memory limits.
• Attention Memory Optimizations - Manages memory allocation for attention mechanisms using slicing to prevent VRAM-related crashes.
• FlashAttention-2 - Integrates FlashAttention-2 via Triton to significantly reduce memory usage and accelerate computation.
• Compute Device Aggregators - Merges multiple compatible compute devices into a single execution unit to enable parallel processing.
• Image Inpainting - Supports image inpainting and outpainting to modify or extend specific image regions.
• Iterative Image Reprocessing - Allows users to reload previously generated images and apply new parameters for iterative editing.
• Triton Kernels - Integrates Triton wheels to compile custom GPU kernels for advanced performance optimizations.
• Hardware-Accelerated Inference - Leverages DirectML to provide hardware-accelerated model inference on Windows devices.
• Image Generation APIs - Exposes a programmable API for triggering generative image and video synthesis from text prompts headlessly.
• Post-Generation Refinements - Implements post-generation passes to refine facial features and fine visual details.
• Intel ARC GPU Acceleration - Launches the image generation server using Intel extensions to utilize ARC graphics hardware.
• ONNX Runtime Inference - Executes image generation models through the ONNX Runtime for cross-platform hardware acceleration.
• Inference Hardware Tuning - Tunes models using quantization and memory offloading to improve inference speed and reduce VRAM usage.
• Model Loading - Implements efficient model loading and binary caching to accelerate the startup of diffusion models.
• Hardware Kernel Selectors - Benchmarks hardware during runtime to automatically select the most efficient execution kernels for the system.
• Model Compilation Optimizers - Uses Triton to enable model compilation for faster execution during inference.
• Graph Compilation Caching - Caches compiled model graphs locally to eliminate the overhead of repeated compilation during startup.
• Olive Model Compression - Utilizes the Olive toolkit to compress and compile models into optimized formats for faster inference.
• Container Weight Persistence - Mounts host directories to the container to ensure model weights and configurations persist across restarts.
• Multi-Architecture Model Compilation - Compiles model graphs using torch.compile and Triton kernels to target specific hardware accelerators.
• OpenVINO Inference Acceleration - Accelerates image generation using OpenVINO-optimized hardware.
• Precision Configuration Tools - Provides utilities for adjusting the numerical precision of models to balance performance and compatibility.
• Programmatic Generation Triggers - Allows users to trigger image generation and processing tasks programmatically through an HTTP API or CLI.
• Weight Quantization - Implements weight quantization to reduce the memory footprint and accelerate the inference of diffusion models.
• Text-Driven Image Editing - Modifies existing images using natural language instructions and specialized editing models.
• Video Generation - Synthesizes short video sequences using generative models from text or image prompts.
• Image Captioning - Uses vision-language models to automatically generate descriptive text captions for images.
• CLI Applications - Provides a command-line interface for triggering generation tasks and managing the server.
• Stateless Application Images - Packages the application into stateless images for different hardware backends to simplify scaling.
• Behavioral Extension Scripts - Supports custom extension development via behavioral scripts that add functionality to the generation pipeline.
• Programmatic Application Control APIs - Provides programmatic application control APIs to manage generative tasks via external automation.
• AI Deployment Containers - Provides pre-configured Docker containers optimized for deploying GPU-accelerated generative AI across different operating systems.
• Cloud Container Deployments - Supports running the application in containers on managed cloud orchestration services with public hostname exposure.
• Container Execution - Provides the ability to launch the application in an isolated container environment with GPU access.
• AI Server Containerization - Packages the generative AI server and API into Docker containers for reproducible deployment.
• Container Image Registry Uploads - Tags and pushes built container images to remote registries for distribution and cloud deployment.
• Containerized Deployments - Ships containerized deployments via Docker to ensure reproducible execution across different hardware backends.
• Hardware-Specific Container Images - Builds portable container images tailored for specific hardware targets like CUDA or ROCm.
• Headless Server Execution - Provides a headless server mode allowing the application to run as an API backend without a graphical interface.
• Translation Layers - Employs a translation layer to enable hardware acceleration on AMD GPUs by converting API calls.
• AI Upscaling - Provides AI-driven resolution upscaling to increase image clarity and detail.
• Local Web Interfaces - Hosts the user interface over HTTP for access via local networks or public URLs.
• Programmatic API Interfaces - Exposes a programmatic API interface for automating image generation and system control without the UI.
• Cross-Platform Compatibility - Supports full feature operation across Linux, Windows, and WSL environments.
• CUDA Driver Wrappers - Implements a wrapper that translates CUDA calls, allowing CUDA-based software to run on AMD hardware.
• Hook-Based Plugin Systems - Utilizes a hook-based plugin system to allow isolated extensions to modify generation and UI pipelines.
• GPU Memory Monitors - Provides real-time tracking of CPU and GPU memory usage to monitor resource consumption during image generation.
• Theme Application & Switching - Provides a system for selecting and applying pre-configured color schemes and visual themes.
• Interface Appearance Customizations - Allows users to modify the visual themes, layouts, colors, and fonts of the user interface.
• UI Theming Engines - Implements a modular engine that switches themes at runtime using a layered CSS override system.

Star history