Building Virtual Worlds with “Digital Twins” – USC Viterbi | School of Engineering

Reconstructing the world in 3D isn’t just a technical challenge — it’s a critical step toward making robotics and autonomous driving systems safer and smarter. This challenge is at the heart of PhD computer science student Jiawei Yang’s research. Yang, who works in USC’s Geometry, Vision, and Learning (GVL) Lab, is the first USC student to receive an NVIDIA Graduate Fellowship awarding outstanding students with up to $60,000 each and an internship preceding the fellowship year. 

Yang is developing self-supervised generative AI models that can reconstruct dynamic 3D scenes quickly and accurately. His groundbreaking work has the potential to advance robotics and autonomous driving systems by creating virtual environments for testing. 

One of the biggest challenges in generative AI today is achieving 3D consistency — ensuring that the geometry and textures in reconstructed scenes remain accurate across frames. Current AI models often generate “hallucinations” during scene reconstruction, leading to potentially incomplete or misleading video outputs, which makes them unsuitable for direct applications in robotics and autonomous driving. 

“My model creates a simulated ‘digital twin’ of our world, making it easier and safer to test robotics and autonomous driving systems.” Jiawei Yang. 

Yang’s research addresses this problem by combining generative AI with cutting-edge 3D computer vision tools. This results in scalable and reliable generative models that eliminate hallucinations and enable the creation of realistic “digital twins” of the surrounding environment. Yang’s model can render new views instantaneously at 200 frames per second, and directly responds to actions taken, allowing real-time, accurate testing of autonomous driving scenarios.

“My model creates a simulated ‘digital twin’ of our world, making it easier and safer to test robotics and autonomous driving systems,” Yang said. “For example, it can reconstruct street views within seconds. The model can also immediately react to new information, such as new camera views of the street, and reconstruct what should be seen.” 

USC’s Geometry, Vision, and Learning (GVL) Lab is dedicated to pushing boundaries in the areas of computer vision, computer graphics, and robotics. The lab’s research is currently centered around the understanding and representation of 3D scenes, the development of foundation models for intelligent robots and vehicles, and the exploration of geometry-inspired deep learning techniques.

“Jiawei’s work exemplifies USC’s commitment to advancing and leveraging AI to solve tough challenges for society,” said Yang’s advisor Yue Wang, an assistant professor of computer science who leads the GVL Lab. “Equally vital to USC’s mission is training the next generation of scientists and engineers like Jiawei to continue this important work.”

Published on February 3rd, 2025

Last updated on February 3rd, 2025