X-PDNet: Accurate Joint Plane Instance Segmentation and Monocular Depth Estimation with Cross-Task Attention and Boundary Correction

Abstract

Segmentation of planar regions from a single RGB image is a particularly important task in the perception of complex scenes. To utilize both visual and geometric properties in images, recent approaches often formulate the problem as a joint estimation of plane instances and dense depth through feature fusion mechanisms and geometric constraint losses. Despite promising results, these methods do not consider cross-task feature distillation and perform poorly at boundary regions. To overcome these limitations, I propose X-PDNet, a framework for the multi-task learning of plane instance segmentation and depth estimation with improvements in the following two aspects. Firstly, I construct the cross-task attention design which promotes early information sharing between multiple tasks for specific task improvements. Secondly, I highlight the current limitations of using the ground truth boundary to develop boundary regression loss, and propose a method that exploits depth information to support precise boundary region segmentation. Finally, I manually annotate more than 3000 images from Stanford 2D-3D-Semantics dataset and make available for evaluation of plane instance segmentation. Through the experiments, my proposed method proves the advantages, outperforming the baseline with large improvement margins in the quantitative results on the ScanNet and the Stanford 2D-3D-S dataset, demonstrating the effectiveness of my proposals.