The advancements in automatic text-to-3D generation have been remarkable. Most existing methods use pre-trained text-to-image diffusion models to optimize 3D representations like Neural Radiance Fields (NeRFs) via latent-space denoising score matching. Yet, these methods often result in artifacts and inconsistencies across different views due to their suboptimal optimization approaches and limited understanding of 3D geometry. Moreover, the inherent constraints of NeRFs in rendering crisp geometry and stable textures usually lead to a two-stage optimization to attain high-resolution details. This work proposes holistic sampling and smoothing approaches to achieve high-quality text-to-3D generation, all in a single-stage optimization. We compute denoising scores in the text-to-image diffusion model's latent and image spaces. Instead of randomly sampling timesteps (also referred to as noise levels in denoising score matching), we introduce a novel timestep annealing approach that progressively reduces the sampled timestep throughout optimization. To generate high-quality renderings in a single-stage optimization, we propose regularization for the variance of z-coordinates along NeRF rays. To address texture flickering issues in NeRFs, we introduce a kernel smoothing technique that refines importance sampling weights coarse-to-fine, ensuring accurate and thorough sampling in high-density regions. Extensive experiments demonstrate the superiority of our method over previous approaches, enabling the generation of highly detailed and view-consistent 3D assets through a single-stage training process.
Image-guided 3D generation
Our paper's main focus is text-to-3d, but we also ran several image-guided 3d generation experiments, with the help of Syncdreamer
![]() |
![]() |
![]() |
![]() |
||||
![]() |
![]() |
Image-to-3D Reconstruction
![]() |
![]() |
![]() |
![]() |
||||
![]() |
![]() |
Text to 3D
We ran extensive text to 3d experiments
![]() |
![]() |
![]() |
![]() |
Chichen itza, aerial view | A christmas tree with donuts as decorations | A baby dragon | A beagle eating a donut |
![]() |
![]() |
![]() |
![]() |
A blue jay standing on a large basket of rainbow macarons | A brightly colored mushroom growing on a log | A cake covered in colorful frosting with | A colorful camping tent in a patch of grass |
![]() |
![]() |
![]() |
![]() |
A dalmation wearing a firemans hat | A dragon-cat hybrid | A drying rack covered in clothes | A flower made out of metal |
![]() |
![]() |
![]() |
![]() |
A goose made out of gold | A lion reading the newspaper | A panda rowing a boat in a pond | A pita bread full of hummus and falafel and vegetables |
![]() |
![]() |
![]() |
![]() |
A plate of delicious tacos | A raccoon astronaut holding his helmet | A red-eyed tree frog | A silver platter piled high with fruits |
![]() |
![]() |
![]() |
![]() |
A squirrel dressed like henry viii king of england | A straw basket with a cobra coming out of it | A toy robot | A yellow schoolbus |
![]() |
![]() |
![]() |
|
An amigurumi motorcycle | An orangutan making a clay bowl on a throwing wheel | An unstable rock cairn in the middle of a stream | |
![]() |
![]() |
||
A baby bunny sitting on top of a stack of pancakes | A ladybug | ||
![]() |
![]() |
||
A broken old clay vessel | A lionfish | ||
![]() |
![]() |
||
An ice cream sundae | A wooden medieval shipping barrel | ||
![]() |
![]() |
||
Robotic bee, high detail | A wooden buddha head | ||
![]() |
![]() |
||
An astronaut is riding a horse | Small saguaro cactus planted in a clay pot | ||
![]() |
![]() |
||
Neuschwanstein Castle, aerial view | A beautifully carved wooden chess piece | ||
![]() |
![]() |
||
A 3d model of an adorable cottage with a thatched roof | A delicious croissant | ||
![]() |
![]() |
||
A DSLR photo of Cthulhu | Donut with blue and yellow sprinkles on top | ||
![]() |
![]() |
||
Michelangelo style statue of dog reading news on a cellphone | Dragon wings and unicorn head hybrid creature | ||
![]() |
![]() |
||
A super mario | A wooden medieval shipping barrel (using another random seed) | ||
![]() |
![]() |
||
A high detailed octopus, 4k | A stack of pancakes covered in maple syrup | ||
![]() |
![]() |
||
A parrot | A peacock on a surfboard | ||
![]() |
![]() |
||
The leaning tower of Pisa, aerial view | A pomeranian dog | ||
![]() |
![]() |
||
A stylized witch pot made of stone or mud with a wide base and a narrow neck, decorated with intricate carvings of runes and symbols, and is filled with a bubbling, green liquid. The pot is surrounded by a cloud of steam* | Pumpkin head zombie, skinny, highly detailed, photorealistic | ||
![]() |
![]() |
||
Gold skull, 4k, highest quality | A pair of pink fluffy slippers | ||
![]() |
![]() |
||
An ice cream sundae | An ice cream sundae (using another random seed) | ||
![]() |
![]() |
||
A car made out of sushi | A car made out of sushi* (using another random seed) | ||
![]() |
![]() |
||
A tarantula, highly detailed | A model of the Eiffel Tower, aerial view | ||
![]() |
![]() |
||
A low-poly tree | A tulip | ||
![]() |
![]() |
||
A blue tulip | A sea turtle | ||
![]() |
|||
A watermelon |