Learning Joint 2D-3D Representations for Depth Completion

    Abstract

    In this paper, we tackle the problem of depth completion from RGBD data. Towards this goal, we design a simple yet effective neural network block that learns to extract joint 2D and 3D features. Specifically, the block consists of two domain-specific sub-networks that apply 2D convolution on image pixels and continuous convolution on 3D points, with their output features fused in image space. We build the depth completion network simply by stacking the proposed block, which has the advantage of learning hierarchical representations that are fully fused between 2D and 3D spaces at multiple levels. We demonstrate the effectiveness of our approach on the challenging KITTI depth completion benchmark and show that our approach outperforms the state-of-the-art.

    Authors

    Yun Chen, Bin Yang, Ming Liang, Raquel Urtasun

    Conference

    ICCV 2019

    Full Paper

    ‘Learning Joint 2D-3D Representations for Depth Completion ‘ (PDF)

    Uber ATG

    Comments
    Previous articleImproving Movement Prediction of Traffic Actors using Off-road Loss and Bias Mitigation
    Next articleOptimization of Swift Protocols
    Yun Chen
    Yun Chen is a graduate student in Pattern Recognition and Intelligent System Laboratory (PRIS Lab) at Beijing University of Posts and Telecommunications, and an AI Resident at Uber ATG Toronto, supervised by Prof. Raquel Urtasun. His research interests include Computer Vision especially Deep Learning.
    Bin Yang
    Bin Yang is a research scientist at Uber ATG Toronto. He's also a PhD student at University of Toronto, supervised by Prof. Raquel Urtasun. His research interest lies in computer vision and deep learning, with a focus on 3D perception in autonomous driving scenario.
    Ming Liang
    Ming Liang is a research scientist at Uber ATG Toronto. Before that he was a senior engineer at Apple SPG. His research interests include neural networks and computer vision.
    Raquel Urtasun
    Raquel Urtasun is the Chief Scientist for Uber ATG and the Head of Uber ATG Toronto. She is also a Professor at the University of Toronto, a Canada Research Chair in Machine Learning and Computer Vision and a co-founder of the Vector Institute for AI. She is a recipient of an NSERC EWR Steacie Award, an NVIDIA Pioneers of AI Award, a Ministry of Education and Innovation Early Researcher Award, three Google Faculty Research Awards, an Amazon Faculty Research Award, a Connaught New Researcher Award, a Fallona Family Research Award and two Best Paper Runner up Prize awarded CVPR in 2013 and 2017. She was also named Chatelaine 2018 Woman of the year, and 2018 Toronto’s top influencers by Adweek magazine