[month] [year]

S Gupta – Learnable HMD Facial De-occlusion

December 2022

Surabhi Gupta received her Master of Science in  Computer Science and Engineering (CSE).  Her research work was supervised by Dr. Avinash Sharma. Here’s a summary of her research work on Learnable HMD Facial De-occlusion for VR Applications in a Person-Specific Setting:

Immersive technologies such as Virtual Reality (VR) and Augmented Reality (AR) are among the fastest growing and fascinating technology today. These technologies promise to provide users with a much better experience using immersive head-mounted displays. Medicine, culture, education, and architecture are some areas that have already taken advantage of this technology. Popular video conferencing platforms such as Microsoft Teams, Zoom, and Google Meet are working to improve user experience by allowing users to use their digital avatars. Nevertheless, they lack immersiveness and realism. Integrating virtual reality platforms in collaborative spaces such as virtual telepresence systems have become quite popular after globalization since it enables multiple users to share the same virtual environment, thus mimicking real-life face-to-face interactions.

For a better immersive experience in virtual telepresence/communication systems, it is essential to recover the entire face, including the portion masked by the headsets. Several methods have been proposed in the literature that deal with this problem in various forms, such as HMD removal and face inpainting. Despite some remarkable explorations, none of these methods promises to provide usable results as expected in virtual reality platforms. Addressing these challenges in the real-world deployment of AR/VR-based applications draws emerging attention. Considering the existing limitations and usability of previous solutions, we explore various research challenges and propose a practical approach to facial de-occlusion/HMD removal for virtual telepresence systems. In addition, we propose an enhancement in eye synthesis using landmarks and also explored temporal coherency in per-frame reconstructions.