stable-dreamfusion

Converts natural language text prompts into 3D models by optimizing a Neural Radiance Field (NeRF) using Score Distillation Sampling (SDS) guidance from Stable Diffusion. The system renders 2D views from the NeRF at each training step, computes diffusion model gradients on those renders conditioned on the text prompt, and backpropagates those gradients through the NeRF parameters to iteratively refine the 3D representation without paired 3D training data.

Generates 3D models from a single reference image by optimizing a NeRF using guidance from the Zero123 model, which performs novel view synthesis. The system renders the NeRF from multiple viewpoints, feeds those renders to Zero123 conditioned on the input image, and uses the diffusion gradients to refine the 3D geometry to be consistent with the reference image across different viewing angles.

Implements automatic checkpoint saving during training, allowing users to resume interrupted training from the latest checkpoint without losing progress. The system saves NeRF model weights, optimizer state, learning rate schedules, and training iteration count at regular intervals. Users can specify checkpoint frequency and directory, and the training loop automatically loads the latest checkpoint on restart.

Provides utilities for preprocessing input images (resizing, normalization, center cropping) and augmenting rendered NeRF outputs (random crops, color jitter, rotation) before feeding to diffusion guidance models. Preprocessing ensures inputs match diffusion model expectations (e.g., 512x512 for Stable Diffusion), while augmentation improves robustness by exposing the NeRF to diverse rendered variations during training.

Provides runtime selection between Taichi (CUDA-free, portable) and CUDA-optimized backends for ray marching and grid encoding computation. Taichi is a domain-specific language for high-performance computing that compiles to CUDA, enabling GPU acceleration without explicit CUDA kernel writing. Users select the backend via configuration, and the system automatically uses the appropriate implementation for ray marching, feature encoding, and other compute-intensive operations.

Implements the Instant-NGP multi-resolution grid encoding scheme to replace vanilla NeRF's positional encoding, enabling 10-100x faster rendering and training. The system uses a hierarchical grid structure with learnable feature vectors at multiple scales (coarse to fine), allowing the network to efficiently represent high-frequency details without dense MLPs. Ray marching queries the grid at each sample point, interpolating features across resolution levels.

Implements a specialized sampling strategy during SDS guidance to mitigate the 'multi-head' problem where the NeRF generates different geometry from different viewpoints. The system samples negative prompts from viewpoints perpendicular to the current rendering direction, encouraging the model to learn consistent 3D structure rather than view-dependent artifacts. This is applied during diffusion guidance by conditioning on both the positive prompt and perpendicular negative views.

Converts the implicit NeRF representation into an explicit mesh (OBJ, PLY) using Differentiable Marching Tetrahedra (DMTet). The system extracts a signed distance field (SDF) from the NeRF's density predictions, applies marching tetrahedra on a tetrahedral grid to generate a mesh, and optionally refines the mesh geometry through additional optimization. The extracted mesh can be textured, edited, or exported to standard 3D software.

Implements efficient ray marching through the 3D scene by sampling points along camera rays and querying the NeRF at each sample point. The system uses adaptive step sizing based on density predictions, skipping empty regions and concentrating samples in high-density areas. Ray marching integrates density and color predictions along the ray to produce final pixel colors, with support for both coarse and fine sampling passes for improved quality.

Provides abstracted NeRF implementations across multiple backends (Instant-NGP, Vanilla NeRF, TCNN, Taichi) with a unified interface, allowing users to select the optimal backend for their hardware and performance requirements. Each backend implements the same forward pass interface but with different underlying representations: grid encoding (Instant-NGP), sinusoidal positional encoding (Vanilla), tiny CUDA neural networks (TCNN), or Taichi-based computation (Taichi). Users specify the backend via command-line arguments.

Provides a modular guidance system supporting multiple diffusion models (Stable Diffusion, Zero123, DeepFloyd IF) through a unified Score Distillation Sampling (SDS) interface. Each guidance module implements the same compute_sds_loss() interface but with model-specific preprocessing, conditioning, and gradient computation. The system loads the appropriate diffusion model based on user selection and applies its gradients to optimize the NeRF.

Provides a graphical user interface (built with Gradio or similar) for text-to-3D and image-to-3D generation without command-line interaction. The GUI accepts text prompts or image uploads, displays real-time or periodic preview renders of the NeRF during training, allows parameter adjustment (guidance scale, learning rate, etc.), and enables one-click mesh export. The interface abstracts away command-line complexity for non-technical users.

Supports rendering the NeRF from multiple camera viewpoints along predefined trajectories (circular orbits, spiral paths, etc.) to generate multi-view image sequences. The system computes camera intrinsics and extrinsics for each viewpoint, performs ray marching from each camera, and outputs a sequence of rendered images. This enables visualization of the 3D model from all angles and generation of training data for downstream tasks.