Compare DreamFusion vs. GET3D

Recent advancements in the realm of text-to-image synthesis have emerged from diffusion models that have been trained on vast amounts of image-text pairs. To successfully transition this methodology to 3D synthesis, it would necessitate extensive datasets of labeled 3D assets alongside effective architectures for denoising 3D information, both of which are currently lacking. In this study, we address these challenges by leveraging a pre-existing 2D text-to-image diffusion model to achieve text-to-3D synthesis. We propose a novel loss function grounded in probability density distillation that allows a 2D diffusion model to serve as a guiding principle for the optimization of a parametric image generator. By implementing this loss in a DeepDream-inspired approach, we refine a randomly initialized 3D model, specifically a Neural Radiance Field (NeRF), through gradient descent to ensure its 2D renderings from various angles exhibit a minimized loss. Consequently, the 3D representation generated from the specified text can be observed from multiple perspectives, illuminated with various lighting conditions, or seamlessly integrated into diverse 3D settings. This innovative method opens new avenues for the application of 3D modeling in creative and commercial fields.

We create a three-dimensional signed distance field (SDF) and a textured field using two latent codes. DMTet is employed to derive a 3D surface mesh from the SDF, and we sample the texture field at the surface points to obtain color information. Our training incorporates adversarial losses focused on 2D images, specifically utilizing a rasterization-based differentiable renderer to produce both RGB images and silhouettes. To distinguish between genuine and generated inputs, we implement two separate 2D discriminators—one for RGB images and another for silhouettes. The entire framework is designed to be trainable in an end-to-end manner. As various sectors increasingly transition towards the development of expansive 3D virtual environments, the demand for scalable tools that can generate substantial quantities of high-quality and diverse 3D content has become apparent. Our research endeavors to create effective 3D generative models capable of producing textured meshes that can be seamlessly integrated into 3D rendering engines, thereby facilitating their immediate application in various downstream uses. This approach not only addresses the scalability challenge but also enhances the potential for innovative applications in virtual reality and gaming.