Skip to footer
Home Research Artificial Intelligence / Machine Learning UPSNet: A Unified Panoptic Segmentation Network

UPSNet: A Unified Panoptic Segmentation Network


In this paper, we propose a unified panoptic segmentation network (UPSNet) for tackling the newly proposed panoptic segmentation task. On top of a single backbone residual network, we first design a deformable convolution based semantic segmentation head and a Mask R-CNN style instance segmentation head which solve these two subtasks simultaneously. More importantly, we introduce a parameter-free panoptic head which solves the panoptic segmentation via pixel-wise classification. It first leverages the logits from the previous two heads and then innovatively expands the representation for enabling prediction of an extra unknown class which helps better resolve the conflicts between semantic and instance segmentation. Additionally, it handles the challenge caused by the varying number of instances and permits back propagation to the bottom modules in an end-to-end manner. Extensive experimental results on Cityscapes, COCO and our internal dataset demonstrate that our UPSNet achieves state-of-the-art performance with much faster inference. Code has been made available at:


Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun


CVPR 2019

Full Paper

‘UPSNet: A Unified Panoptic Segmentation Network’ (PDF)

Uber ATG

Previous article DeepSignals: Predicting Intent of Drivers Through Visual Attributes
Next article Building Together: When Research Went Viral at Uber
Yuwen Xiong
Yuwen Xiong is a graduate student in Machine Learning Group at the University of Toronto, and a Research Scientist at Uber ATG Toronto, both supervised by Prof. Raquel Urtasun. Before that he received his bachelor degree in Computer Science from Zhejiang University in June 2018. His research interests include Computer Vision and Machine Learning, especially Deep Learning.
Renjie Liao
Renjie Liao is a PhD student in Machine Learning Group, Department of Computer Science, University of Toronto, supervised by Prof. Raquel Urtasun and Prof. Richard Zemel. He is also a Research Scientist in Uber Advanced Technology Group Toronto. He is also affiliated with Vector Institute. He received M.Phil. degree from Department of Computer Science and Engineering, Chinese University of Hong Kong, under the supervision of Prof. Jiaya Jia. He got B.Eng. degree from School of Automation Science and Electrical Engineering in Beihang University (former Beijing University of Aeronautics and Astronautics).
Rui Hu
Rui Hu is a research engineer at Uber ATG Toronto. Before that he was a staff engineer at Qualcomm and senior engineer at Apple SPG. His research interests include neural networks, computer vision and GPU optimization.
Min Bai
Min Bai is a research scientist at Uber ATG Toronto. Before that, he was a wireless systems engineer at Apple. He has an undergraduate degree in electrical engineering from the University of Waterloo. His research interest includes various perception tasks such as segmentation, point cloud processing, online mapping.
Ersin Yumer
Ersin Yumer is a Staff Research Scientist, leading the San Francisco research team within Uber ATG R&D. Prior to joining Uber, he led the perception machine learning team at Argo AI, and before that he spent three years at Adobe Research. He completed his PhD studies at Carnegie Mellon University, during which he spent several summers at Google Research as well. His current research interests lie at the intersection of machine learning, 3D computer vision, and graphics. He develops end-to-end learning systems and holistic machine learning applications that bring signals of the visual world together: images, point clouds, videos, 3D shapes and depth scans.
Raquel Urtasun
Raquel Urtasun is the Chief Scientist for Uber ATG and the Head of Uber ATG Toronto. She is also a Professor at the University of Toronto, a Canada Research Chair in Machine Learning and Computer Vision and a co-founder of the Vector Institute for AI. She is a recipient of an NSERC EWR Steacie Award, an NVIDIA Pioneers of AI Award, a Ministry of Education and Innovation Early Researcher Award, three Google Faculty Research Awards, an Amazon Faculty Research Award, a Connaught New Researcher Award, a Fallona Family Research Award and two Best Paper Runner up Prize awarded CVPR in 2013 and 2017. She was also named Chatelaine 2018 Woman of the year, and 2018 Toronto’s top influencers by Adweek magazine