# lllyasviel/controlnet-v1-1-nightly

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/lllyasviel-controlnet-v1-1-nightly).**

5,156 stars · 407 forks · Python

## Links

- GitHub: https://github.com/lllyasviel/ControlNet-v1-1-nightly
- awesome-repositories: https://awesome-repositories.com/repository/lllyasviel-controlnet-v1-1-nightly.md

## Description

This project is a neural network extension for Stable Diffusion that provides spatial control and geometric consistency for text-to-image generation. It functions as an image structure controller and conditioning tool, enabling the use of external inputs to guide the layout and geometry of generated imagery.

The framework is distinguished by its ability to transform input images into structural guides through various preprocessors. These include the extraction of depth maps, normal maps, and human pose landmarks, as well as the detection of Canny edges, anime lineart, and straight architectural lines. It also supports semantic segmentation to define object placement via colored masks and converts hand-drawn scribbles into detailed images.

Beyond basic conditioning, the project covers image editing and upscaling through tiled detail refinement and inpainting. It provides tools for custom diffusion model training, including dataset annotation and content shuffle preprocessing. Performance is managed via GPU memory optimizations such as sliced attention to reduce resource consumption during the sampling process.

## Tags

### Artificial Intelligence & ML

- [Spatial Conditioning Controllers](https://awesome-repositories.com/f/artificial-intelligence-ml/spatial-processing-operations/spatial-processing-operations/spatial-conditioning-controllers.md) — Injects structured image data like depth and edge maps as conditioning signals to guide the denoising process.
- [Diffusion Structural Control](https://awesome-repositories.com/f/artificial-intelligence-ml/diffusion-structural-control.md) — Provides a framework for guiding diffusion model output using spatial constraints like depth maps and semantic segmentation.
- [Visual Landmark Extractors](https://awesome-repositories.com/f/artificial-intelligence-ml/feature-extractors/visual-landmark-extractors.md) — Identifies human body and face landmarks from photographs to control character posture in generated images. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/github_docs/annotator.md))
- [Generative Pose Control](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-pose-control.md) — Constrains character poses using coordinates for the body, hands, and face in generated images. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [Image Composition Controls](https://awesome-repositories.com/f/artificial-intelligence-ml/image-composition-controls.md) — Provides frameworks for managing spatial layout, pose, and arrangement using structural maps like depth and edges.
- [Canny Edge Conditioning](https://awesome-repositories.com/f/artificial-intelligence-ml/image-generation-models/conditional-image-generation/canny-edge-conditioning.md) — Uses Canny edge detection maps to strictly constrain the boundaries and shapes of generated objects. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [Depth Map Conditioning](https://awesome-repositories.com/f/artificial-intelligence-ml/image-generation-models/conditional-image-generation/depth-map-conditioning.md) — Employs depth information from rendering engines or preprocessors to maintain spatial layout during generation. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [Edge Map Conditioning](https://awesome-repositories.com/f/artificial-intelligence-ml/image-generation-models/conditional-image-generation/edge-map-conditioning.md) — Provides spatial control using soft-edge maps to define boundaries and remove visual artifacts during generation. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [Structural Conditioning](https://awesome-repositories.com/f/artificial-intelligence-ml/image-generation-models/conditional-image-generation/structural-conditioning.md) — Transforms input images into structural guides via preprocessors to direct text-to-image generation.
- [Semantic Segmentation](https://awesome-repositories.com/f/artificial-intelligence-ml/semantic-segmentation.md) — Classifies image regions into semantic categories to define object placement and layout in generated images. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/github_docs/annotator.md))
- [Structural Image Generation](https://awesome-repositories.com/f/artificial-intelligence-ml/structural-image-generation.md) — Enforces precise architectural and geometric consistency in generated images using line detection and depth information.
- [Adapter-Based Conditioning](https://awesome-repositories.com/f/artificial-intelligence-ml/structural-image-generation/adapter-based-conditioning.md) — Injects structural constraints via line drawings using adapter-based conditioning to define output structure. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [Line-Based Geometric Constraints](https://awesome-repositories.com/f/artificial-intelligence-ml/structural-image-generation/line-based-geometric-constraints.md) — Employs straight line detection to guide the creation of architecturally consistent environments. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [Block-Wise Attention](https://awesome-repositories.com/f/artificial-intelligence-ml/attention-mechanisms/flash-attention-implementations/block-wise-attention.md) — Implements block-wise attention to reduce GPU memory consumption during the diffusion sampling process.
- [Image Inpainting](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-ai-resources/diffusion-visual-models/generative-ai-pipelines/text-to-image-generators/image-inpainting.md) — Fills masked image regions using structural conditioning to ensure generated content matches the surrounding scene.
- [Generative Upscalers](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-upscalers.md) — Increases image resolution and detail using a tile-based generative approach to maintain spatial consistency. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/gradio_tile.py))
- [Sequential Detail Refinement](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-upscalers/sequential-detail-refinement.md) — Generates fine-grained details in image tiles through sequential refinement to enable high-resolution upscaling. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [GPU Memory Optimizers](https://awesome-repositories.com/f/artificial-intelligence-ml/gpu-memory-optimizers.md) — Optimizes VRAM usage during model execution through techniques like sliced attention to reduce GPU memory consumption. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/share.py))
- [Architectural Generation](https://awesome-repositories.com/f/artificial-intelligence-ml/image-generation/architectural-generation.md) — Maintains geometric consistency in architectural imagery using specialized straight-line detection.
- [Text-Instruction Editors](https://awesome-repositories.com/f/artificial-intelligence-ml/image-generation/image-editing/text-instruction-editors.md) — Modifies existing images based on natural language instructions and descriptive prompts. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [Memory Saving Modes](https://awesome-repositories.com/f/artificial-intelligence-ml/inference-memory-optimizations/memory-saving-modes.md) — Offers configurable memory saving modes that trade computation speed for reduced peak memory usage during image generation. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/config.py))
- [Scribble-to-Image Generation](https://awesome-repositories.com/f/artificial-intelligence-ml/scribble-to-image-generation.md) — Transforms hand-drawn scribbles or edge maps into detailed images using them as spatial conditioning. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/gradio_scribble.py))
- [Scribble Conditioning](https://awesome-repositories.com/f/artificial-intelligence-ml/structural-image-generation/scribble-conditioning.md) — Transforms hand-drawn or synthesized scribbles into detailed images while preserving the original sketch layout. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [Detail Refinement](https://awesome-repositories.com/f/artificial-intelligence-ml/tiled-processing/image-tiling/detail-refinement.md) — Processes images in small overlapping blocks to generate high-resolution details while managing GPU memory.
- [Tiled Upscaling](https://awesome-repositories.com/f/artificial-intelligence-ml/tiled-processing/image-tiling/tiled-upscaling.md) — Increases image resolution and adds fine-grained detail using a tile-based refinement process.
- [Weight-Locked Architectures](https://awesome-repositories.com/f/artificial-intelligence-ml/weight-locked-architectures.md) — Utilizes a mirrored weight-locked architecture to preserve pre-trained knowledge while learning new structural controls.
- [Zero-Convolutional Layers](https://awesome-repositories.com/f/artificial-intelligence-ml/zero-convolutional-layers.md) — Uses zero-initialized convolutional layers to inject spatial control signals without disrupting the frozen pre-trained model.

### Part of an Awesome List

- [Boundary Constraints](https://awesome-repositories.com/f/awesome-lists/ai/edge-detection/boundary-constraints.md) — Implements gradient-based Canny edge detection to enforce strict structural outlines in generated images.
- [Pose Synthesis](https://awesome-repositories.com/f/awesome-lists/ai/human-image-and-video-generation/pose-synthesis.md) — Defines exact postures and gestures of characters using body and face landmarks.
- [Semantic Image Synthesis](https://awesome-repositories.com/f/awesome-lists/ai/image-generation-and-synthesis/semantic-image-synthesis.md) — Generates photorealistic images using colored semantic masks to assign object categories to specific regions. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [Straight Line Detections](https://awesome-repositories.com/f/awesome-lists/ai/edge-detection/straight-line-detections.md) — Implements straight line detection to maintain geometric and architectural consistency in generated images. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/gradio_mlsd.py))
- [Stable Diffusion Ecosystem](https://awesome-repositories.com/f/awesome-lists/ai/stable-diffusion-ecosystem.md) — Extends the Stable Diffusion framework by adding spatial control capabilities via specialized conditioning models.

### Data & Databases

- [Layout Masks](https://awesome-repositories.com/f/data-databases/enum-definitions/enum-label-mappings/category-identifier-mappings/semantic-to-visual-mappings/layout-masks.md) — Enables precise spatial placement of scene elements by mapping specific colors to object categories.

### Graphics & Multimedia

- [Structural Map Generators](https://awesome-repositories.com/f/graphics-multimedia/structural-map-generators.md) — Generates distance-based depth maps from images to provide spatial structural guidance for image generation. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/github_docs/annotator.md))
- [Pose Guidance](https://awesome-repositories.com/f/graphics-multimedia/video-frame-processing/real-time-model-inference-on-frames/human-pose-detections/pose-guidance.md) — Provides spatial guidance via coordinate-based landmark maps to control the skeletal structure and posture of humans.
- [Anime Lineart Extractions](https://awesome-repositories.com/f/graphics-multimedia/anime-lineart-extractions.md) — Provides the ability to extract line art from anime images to guide the structure of generated variations. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/gradio_lineart_anime.py))
- [Edge Extraction Utilities](https://awesome-repositories.com/f/graphics-multimedia/edge-extraction-utilities.md) — Provides utilities to extract structural edges and line drawings from photographs to serve as generation guides. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/github_docs/annotator.md))
- [AI Upscaling](https://awesome-repositories.com/f/graphics-multimedia/image-editing-processing/image-enhancement-tools/ai-upscaling.md) — Increases image resolution and adds fine-grained details using a tile-based AI upscaling approach.

### User Interface & Experience

- [Normal Map Conditioning](https://awesome-repositories.com/f/user-interface-experience/texture-mapping/normal-maps/normal-map-conditioning.md) — Uses surface normal maps to define the orientation of objects and surfaces in generated imagery. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly#readme))
- [Normal Map Extractions](https://awesome-repositories.com/f/user-interface-experience/texture-mapping/normal-maps/normal-map-extractions.md) — Detects surface orientations from images to guide the lighting and geometry of generated content. ([source](https://github.com/lllyasviel/ControlNet-v1-1-nightly/blob/main/github_docs/annotator.md))