lllyasviel/Omost
Omost
Omost is a system of software components designed for iterative image refinement, regional layout control, and the optimization of text-to-image embedding processes. It functions as a diffusion model layout controller and an engine that uses large language models to generate executable code for precise control over image composition.
The project features a conversational image editor that allows for the refinement of visual content through natural language instructions and automated code execution. It distinguishes itself through a text embedding optimizer that organizes sub-prompts into tree graphs to prevent semantic truncation during text encoding.
The toolset covers regional image generation and complex image composition using grid-based layouts and regional attention guidance. It includes capabilities for visual element composition based on depth and color, as well as the integration of stylistic metadata to refine the overall atmospheric output.
Features
- Code-Driven Layout Generation - Uses an LLM to generate executable code for precise control over image bounding boxes and composition.
- Regional - Manipulates attention scores to ensure specific text prompts only affect designated image areas.
- Diffusion Layout Controllers - Defines regional bounding boxes and attention scores to guide the generation process of diffusion models.
- Graph-Based Prompt Organization - Structures independent descriptive concepts into a graph to merge them into cohesive prompts.
- Image Composition Controls - Uses large language models to generate executable code for precise control over image layout and composition.
- Conversational Editing Interfaces - Refines image composition and details through a chat interface instead of rewriting prompts.
- Regional Prompting - Controls the placement and appearance of specific objects using bounding boxes and regional attention.
- Semantic Embedding Merging - Implements a greedy merging strategy for sub-prompts to prevent semantic truncation during text encoding.
- Precise Visual Layout Control - Ensures visual elements appear in exact intended positions using a grid system of global and local descriptions.
- Embedding Optimization Processes - Organizes sub-prompts into tree graphs to prevent semantic truncation during text encoding.
- Embedding Optimizers - Provides a greedy merging strategy for sub-prompts to ensure coherent text encoding and prevent semantic truncation.
- Grid-Based Image Layouts - Uses a discretized grid system to assign global and local descriptions to specific bounding boxes.
- Conversational Image Editors - Provides a chat interface for iteratively refining visual content through natural language and automated code execution.
- Generative - Provides a grid-based coordinate system to map global and local descriptions to specific image areas.
- Latent Layout Mappings - Organizes image components by depth and color to create initial latent maps for diffusion models.
- Image Editing - Allows users to refine generated visual content through iterative, chat-based adjustments to the image composition.
- Prompt Optimizers - Optimizes descriptive concepts using structured graphs and embedding merges to prevent semantic truncation.
- Code-to-Image Composition - Converts natural language prompts into executable code for a virtual canvas agent to arrange complex visual content.
- Prompt Graph Organizers - Implements a prefix tree system to organize independent descriptive concepts into cohesive prompts via specific traversal paths.
- Latent Element Composition - Organizes image components by relative depth and color to create layout maps for use as initial latents.
- Image Composition - Arranges multiple visual elements on a virtual canvas using code and layout maps to create structured scenes.
