Deep Depth Completion of a Single RGB-D Image

Abstract

The goal of this work is to complete the depth channel of an RGB-D image. Commodity-grade depth cameras often fail to sense depth for shiny, bright, transparent, and distant surfaces. To address this problem, we train a deep network that takes an RGB image as input and predicts dense surface normals and occlusion boundaries. Those predictions are then combined with raw depth observations provided by the RGB-D camera to solve for depths for all pixels, including those missing in the original observation. This method was chosen over others (e.g., inpainting depths directly) as the result of extensive experiments with a new depth completion benchmark dataset, where holes are filled in training data through the rendering of surface reconstructions created from multiview RGB-D scans. Experiments with different network inputs, depth representations, loss functions, optimization methods, inpainting methods, and deep depth estimation networks show that our proposed approach provides better depth completions than these alternatives.

Paper

• Yinda Zhang, Thomas Funkhouser.
  Deep Depth Completion of a Single RGB-D Image
  Computer Vision and Pattern Recognition (CVPR 2018)
  [Paper]   [Supplementary Materials]   [spotlight]
  

Code and Dataset

• Please check our repository for how to access our data and run our codes.
  [Git Repository]


• Our data is created from SUNCG-RGBD, Matterport3D, and ScanNet. If you are interested in using any part of our data, you must obtain the access to the corresponding original datasets that they are generated from. Unfortunately, SUNCG-RGBG, Matterport3D, and ScanNet are using different terms of usage, and hence you may need to get accesses to all of them in order to have a full access to our dataset. Please send an email to the dataset orgnizer(s) to confirm your agreement and cc Yinda Zhang (yindaz guess cs dot princeton dot edu). You will get download links for the data from the dataset that you got approval.


• Dataset Snapshots

Result

Sensor Color	Sensor Depth	Our Completion Result	Sensor Point Cloud	Our Result Point Cloud