RODIN Reviews

Recent breakthroughs in text to image synthesis were driven by diffusion models that have been trained on billions and billions of image/text pairs. To adapt this approach to 3D, large datasets of labeled 3D asset and efficient architectures for denoising 3D data would be required. In this work, these limitations are circumvented by using a 2D text to image diffusion model that has been pre-trained. We introduce a loss that is based on probability densities distillation, which allows a 2D diffusion to be used as a prior in the optimization of a parameterized image generator. We optimize a randomly initialized 3D model, a Neural Radiance field, or NeRF, via gradient descent using this loss. This allows us to achieve low losses in 2D renderings of the model from random angles. The 3D model can be viewed at any angle and lit by any illumination. It can also be composited into any 3D environment.