ashawkey/stable-dreamfusion
Stable Dreamfusion
This project is a diffusion-based 3D generator and image-to-3D reconstruction system. It translates natural language descriptions or two-dimensional images into three-dimensional assets using neural radiance fields and diffusion models.
The system utilizes score-distillation sampling and diffusion-based guidance to refine 3D shapes without requiring 3D training data. It includes specialized tools for transforming neural representations into exportable meshes with texture and material data, as well as a pipeline for iterative optimization of geometry and textures.
The project covers a broad range of geometry processing and media capabilities, including mesh cleaning, complexity reduction, and point cloud reconstruction. It also provides utilities for image preprocessing, such as depth and normal estimation, background removal, and the rendering of orbiting videos to evaluate generated assets.
The project provides environment containerization to ensure consistent execution across different hardware setups.
Features
- Text-to-3D Generators - Translates natural language descriptions into three-dimensional assets using diffusion models and neural radiance fields.
- Text-to-Mesh Generation - Produces three-dimensional geometry by translating textual descriptions into assets using diffusion models.
- Diffusion-Based 3D Generators - Translates text or image prompts into three-dimensional assets using diffusion models and neural radiance fields.
- Diffusion-Based 3D Guidance - Uses a pre-trained image generation model as a loss function to refine 3D shapes without 3D training data.
- Image-Conditioned 3D Generation - Uses reference images and depth maps as geometric priors to guide the synthesis of 3D representations.
- Neural Radiance Field Synthesizers - Represents 3D scenes as a continuous volumetric function mapping spatial coordinates to color and density.
- Score Distillation Sampling - Optimizes 3D geometry by calculating gradients from a 2D diffusion model to align renders with prompts.
- 3D Reconstruction - Transforms two-dimensional images into three-dimensional mesh representations through image-conditioned generation.
- Generative Text-to-3D Pipelines - Provides a workflow for generating 3D geometry and textures from natural language descriptions through iterative optimization.
- Volumetric Rendering Engines - Computes 2D image pixels by integrating light transport along camera rays through a learned 3D density field.
- SDS Loss Optimization - Refines 3D representations by calculating gradients between predicted and added noise to align renders with models.
- Image-to-Mesh Generation - Transforms two-dimensional image inputs into three-dimensional mesh representations using image-conditioned generation.
- 3D Asset Exporters - Provides tools to convert internal 3D scene representations into industry-standard file formats for use in external software.
- 3D Asset Pipelines - Refines generated meshes by reducing complexity, cleaning non-manifold edges, and increasing geometric resolution.
- Mesh Optimization - Decreases the number of faces in a mesh through edge collapse to reach a target face count.
- Geometric Resolution Refinement - Increases the detail of generated meshes by fine-tuning existing checkpoints to produce sharper geometric results.
- Mesh-to-Video Renderers - Generates video sequences from meshes by defining object centering, environment parameters, and transformation vectors.
- Volumetric Mesh Exporters - Extracts polygonal meshes from neural volumetric representations for use in external 3D software.
- Mesh Processing Tools - Removes unreferenced vertices and duplicate faces while repairing non-manifold edges to improve model quality.
- Volumetric Mesh Extraction - Extracts a polygonal surface mesh from a dense volumetric grid by identifying boundaries of specific density values.
- 360 Orbit Renderers - Generates a 360-degree video orbiting the created object to evaluate the result from all angles.
- Geometric Artifact Reduction - Applies perpendicular negation techniques to prevent the generation of duplicate heads or mirrored geometry.
- Generative AI Models - Open source implementation for text-to-3D generation.
- Stable Diffusion Ecosystem - Text-to-3D generation powered by diffusion models.
