DeepDR: Deep Structure-Aware RGB-D Inpainting for Diminished Reality

Diminished reality (DR) refers to the removal of real objects from the environment by virtually replacing them with their background. Modern DR frameworks use inpainting to hallucinate unobserved regions. While recent deep learning-based inpainting is promising, the DR use case is complicated by the need to generate coherent structure and 3D geometry (i.e., depth), in particular for advanced applications, such as 3D scene editing. In this paper, we propose DeepDR, a first RGB-D inpainting framework fulfilling all requirements of DR: Plausible image and geometry inpainting with coherent structure, running at real-time frame rates, with minimal temporal artifacts. Our structure-aware generative network allows us to explicitly condition color and depth outputs on the scene semantics, overcoming the difficulty of reconstructing sharp and consistent boundaries in regions with complex backgrounds. Experimental results show that the proposed framework can outperform related work qualitatively and quantitatively.

Tunnel Surface 3D Reconstruction from Unoriented Image Sequences

The 3D documentation of the tunnel surface during construction requires fast and robust measurement systems. In the solution proposed in this paper, during tunnel advance a single camera is taking pictures of the tunnel surface from several positions. The recorded images are automatically processed to gain a 3D tunnel surface model. Image acquisition is realized by the tunneling/advance/driving personnel close to the tunnel face (= the front end of the advance). Based on the following fully automatic analysis/evaluation, a decision on the quality of the outbreak can be made within a few minutes. This paper describes the image recording system and conditions as well as the stereo-photogrammetry based workflow for the continuously merged dense 3D reconstruction of the entire advance region. Geo-reference is realized by means of signalized targets that are automatically detected in the images. We report on the results of recent testing under real construction conditions, and conclude with prospects for further development in terms of on-site performance.