Farm3D

Farm3D Use Cases

Farm3D learns an articulated object category entirely from "free" virtual supervision from a 2D diffusion-based image generator.

The Farm3D framework employs an image generator, such as Stable Diffusion, to produce training data for learning a reconstruction network from the ground up.

The diffusion model used in Farm3D is incorporated as a scoring mechanism to further improve learning.

Farm3D's monocular reconstruction network is capable of generating controllable 3D assets from a single input image, whether real or generated, in a matter of seconds.

Farm3D reconstructs the shape of a wide range of categories to their fine details such as legs and ears despite not being trained on any real images.

Farm3D enables the generation of controllable 3D assets from either a real image or an image synthesized using Stable Diffusion.

Farm3D allows for adjusting lighting, swapping textures between models of the same category, and animating the shape of the generated 3D assets.

Farm3D supports animation of the generated 3D assets based on a single input image.

Farm3D allows for relighting of the generated 3D assets based on a single input image.

Farm3D enables texture swapping between models of the same category based on a single input image.

Farm3D's approach involves using Stable Diffusion to generate virtual views of an object category, which are then used to train a monocular articulated object reconstruction model.

During training, synthetic instance views are sampled and "critiqued" by Stable Diffusion to guide the learning process.

Farm3D's single-view 3D reconstruction is capable of reconstructing the shape of various categories with fine details, even without being trained on real images.