Workshop on IMmersive Mixed and Virtual Environment Systems 

The 15th International Workshop on IMmersive Mixed and Virtual Environment Systems (MMVE) was held the 10th o June in Vancouver, Canada, co-located with ACM Multimedia Systems Conference (MMSys2023).

In this workshop, Teresa Hernando, researcher of the Grupo de Tratamiento de Imágenes, presented the contribution “Real-Time Layered View Synthesis for Free-Viewpoint Video from Unreliable Depth Information”.

This work presents a novel approach for the generation in real time of synthetic views for free-viewpoint video. Our system is based on purely passive stereo cameras which, under the constraints of real-time operation yield unreliable depth maps, especially in the background areas. To solve this issue, we propose a layered synthesis algorithm that combines offline and online sources of colour and depth information to increase the quality of the synthesized image. Foreground objects are synthesized using online depth and colour information. The background, however, is assumed to be stationary so its geometry can be reconstructed offline. However, foreground objects affect the colour of the background, so to render the background we carefully mix online and offline colour information. Finally, foreground and background are combined to form the final image.

This work has been partially supported by project “XReco” (HORIZON- IA-101070250) funded by the European Union, by project “SARAOS” (PID2020-115132RB) funded by MCIN/AEI/10.13039/501100011033 of the Spanish Government, and by project UNICO-5G I+D TSI- 063000-2021-80 (DISRADIO – Pilotos) funded by the Ministry of Economic Affairs and Digital Transformation of the Spanish Government and the NextGenerationEU (Recovery, Transformation and Resilience Plan - PRTR).

Link to the paper: https://dl.acm.org/doi/abs/10.1145/3592834.3592881