NVIDIA is showcasing its latest advancements in visual AI at the Computer Vision and Pattern Recognition (CVPR) conference in Seattle. These advancements cover a range of areas, including custom image creation, 3D scene editing, understanding visual language, and improving the perception systems of self-driving cars.

Jan Kautz, VP of learning and perception research at NVIDIA, emphasized the significance of artificial intelligence, particularly generative AI, as a key technological breakthrough. He highlighted that NVIDIA’s research is pushing the limits of what AI can achieve, from enhancing tools for professional creators to developing software for next-generation autonomous vehicles.

Among the more than 50 research projects presented by NVIDIA, two papers are finalists for the Best Paper Awards at CVPR. One of these papers focuses on the training processes of diffusion models, while the other discusses high-definition mapping for self-driving cars.

NVIDIA also won the CVPR Autonomous Grand Challenge’s End-to-End Driving at Scale track, standing out among over 450 global entries. This achievement highlights NVIDIA’s leading role in using generative AI for comprehensive self-driving vehicle models and earned them an Innovation Award from CVPR.

Key projects include:

1. JeDi: A technique allowing creators to quickly customize diffusion models to depict specific objects or characters using only a few reference images, simplifying the usually time-consuming fine-tuning process.
2. FoundationPose: A foundational model that can instantly recognize and track the 3D positions of objects in videos without needing separate training for each object. This model set a new performance benchmark and could have significant applications in augmented reality and robotics.
3. NeRFDeformer: A method for editing 3D scenes captured by a Neural Radiance Field (NeRF) using a single 2D image, eliminating the need to manually reanimate or completely recreate the scene. This could make 3D scene editing easier for use in graphics, robotics, and digital twins.

In collaboration with MIT, NVIDIA developed VILA, a new series of vision language models. VILA excels in understanding images, videos, and text, and has advanced reasoning abilities that enable it to comprehend internet memes by integrating visual and linguistic information.

NVIDIA’s research in visual AI impacts various industries, with more than a dozen papers on innovative methods for improving autonomous vehicle perception, mapping, and planning. Sanja Fidler, VP of NVIDIA’s AI Research team, is presenting on the potential of vision language models in self-driving cars.

NVIDIA’s research at CVPR illustrates how generative AI can empower creators, speed up automation in manufacturing and healthcare, and drive advancements in autonomy and robotics.