Meta Reality Labs researchers have developed a prototype VR headset with a custom accelerator chip built specifically for processing artificial intelligence, making it possible to render the company’s photorealistic Codec Avatars on a standalone headset.
Before the company changed its name, Meta had been pushing the Codec Avatars project, which aims to make near-realistic avatars in VR a reality. Using a combination of on-device sensors (such as eye-tracking and mouth tracking) and artificial intelligence processing, the system creates detailed animations for the user in real-time in a realistic manner.
Earlier versions of the Codec Avatars project required the use of NVIDIA’s Titan X GPU for rendering support, making it impossible for mobile standalone headsets like Meta’s latest Quest 2 to drive those needs.
So the company has begun researching how to implement Codec Avatars on low-power standalone headsets, as evidenced by a paper presented at the 2022 IEEE CICC conference last month. In the paper, Meta revealed that it created a custom chip fabricated on a 7-nanometer process as an accelerator specifically for Codec Avatars.
According to the researchers, the chip is far from off-the-shelf. The team designed it with an important part of the Codec Avatars processing pipeline in mind, specifically analyzing incoming eye-tracking images and generating the data needed to model Codec Avatars. The chip’s footprint is only 1.6mm².
“The test chip fabricated at the 7nm technology node features a neural network (NN) accelerator consisting of a 1024 product (MAC) array, 2MB of on-chip SRAM, and a 32-bit RISC-V CPU,” the researchers wrote. In turn, they also rebuilt parts of the Codec Avatars AI model to take advantage of the chip’s specific architecture.
“By reconstructing a convolutional [neural network]-based eye gaze extraction model and tailoring it to hardware, the entire model is suitable for use on-chip to mitigate system-level energy and latency costs of off-chip memory access,” the researchers said. By efficiently accelerating convolution operations at the circuit level, the proposed prototype [chip] achieves 30 frames per second performance and low power consumption at a low form factor.”
By accelerating intensive portions of Codec Avatars workloads, the chip not only speeds up the process but also reduces the power and heat required. Thanks to the chip’s custom design, it was able to do this more efficiently than a general-purpose CPU, which then informed the re-architected software design of Codec Avatars’ eye-tracking components.
But the headset’s general-purpose CPU (in this case, the Quest 2’s Snapdragon XR2 chip) isn’t sitting idle either. While the custom chip handles part of the encoding process for Codec Avatars, XR2 manages the decoding process and renders the actual visuals of the avatar.
The work must be quite multidisciplinary, as the paper mentions 12 researchers, all from Meta’s Reality Lab. H. Ekin Sumbul, Tony F. Wu, Yuecheng Li, Syed Shakib Sarwar, William Koven, Eli Murphy-Trotzky, Xingxing Cai, Elnaz Ansari, Daniel H. Morris, Huichu Liu, Doyun Kim, and Edith Beigne.
Impressively, Meta’s Codec Avatars can run on standalone headsets, even if a special chip is required. But one thing we don’t know is how the visual rendering of the avatar is handled. The user’s underlying scans are highly detailed and may be too complex to render fully on the Quest 2. It’s unclear how much of the “realistic” part of Codec Avatars is preserved in this case, even though all the base parts are there to drive the animation.