AI-powered ‘DyLiN’ System Can Turn 2D Images Into Dynamic 3D Models

Carnegie Mellon University (CMU) and Fujitsu researchers have pioneered a method of converting a 2D image into a three-dimensional one. This is a challenging task because it requires understanding how to handle deformations and changes that occur when going from 2D to 3D.

The researchers’ Dynamic Light Field Network (DyLiN) method uses artificial intelligence techniques to tackle this difficulty. It learns a deformation field, which is a way to describe how the shape of an object changes, from the input images to a canonical representation of the object. This canonical representation helps to handle any abrupt changes or discontinuities in the object’s shape.

The researchers also introduced another model called CoDyLiN, which builds upon DyLiN by allowing users to control certain attributes of the 3D structure. This gives more flexibility and control in the generation process.

To train these models, the researchers used a technique called knowledge distillation. They took advantage of existing models that were already trained on dynamic radiance fields, which are representations of 3D scenes with changing lighting conditions, to guide and improve the training of DyLiN and CoDyLiN.

According to the researchers, their method matched the visual fidelity of other state-of-the-art methods while being 25 to 71 times faster in terms of computation times. “The strides made by DyLiN and CoDyLiN mark an exciting progression toward real-time volumetric rendering and animation,” said lead CMU researcher Laszlo Jeni.

Both DyLiN and CoDyLiN offer applications such as modelling 3D human faces based on 2D images. That points to further advancement in the engineering of realistic deepfakes, including convincing 3D avatars that could potentially be animated in real time.

Source: Vision-Spectra

–

July 11, 2023 – by the FindBiometrics Editorial Team