Wide-baseline Omnidirectional Depth Estimation using Convolutional Neural Network

Abstract

Autonomous robots and drones operating in the real-world need to sense the surrounding environmental structure in 3D. In congested situation such as crowded downtown, all-around depth as well as front depth is needed to detect obstacles and avoid them. In this thesis, we propose a novel omnidirectional stereo algorithm which estimates the dense depth map from the fisheye images using a deep convolutional neural network. The capture system consists of multiple cameras mounted on a wide-baseline rig with ultra-wide field of view (FOV) lenses. We introduce the calibration algorithm for the extrinsic parameters of the rig by using the bundle adjustment with checkerboard images. Instead of computing depth maps from multiple sets of rectified images and stitching them, our stereo algorithm directly generates one dense omnidirectional depth map with full 360° coverage at the global rig coordinate system. The proposed network is designed to output the matching cost volume from the warped images in the sphere sweeping method. The cost volume is aggregated by Semi-Global Matching, and the final depth map is determined by taking the minimum indices of the aggregated cost volume. Furthermore, we render realistic synthetic urban datasets to train the deep neural network and test the entire system. The experiments using the synthetic and real-world datasets show that our algorithm outperforms the conventional depth estimation methods and generate highly accurate depth maps.