Skip to footer
Home Research Artificial Intelligence / Machine Learning MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving

MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving



While most approaches to semantic reasoning have focused on improving performance, in this paper we argue that computational times are very important in order to enable real time applications such as autonomous driving. Towards this goal, we present an approach to joint classification, detection and semantic segmentation using a unified architecture where the encoder is shared amongst the three tasks. Our approach is very simple, can be trained end-to-end and performs extremely well in the challenging KITTI dataset. Our approach is also very efficient, allowing us to perform inference at more then 23 frames per second. Training scripts and trained weights to reproduce our results can be found here:


Martin Teichmann, Michael Weber, Marius Zöllner, Roberto Cipolla, Raquel Urtasun


IV 2018

Full Paper

‘MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving’ (PDF)

Uber ATG

Previous article COTA: Improving the Speed and Accuracy of Customer Support through Ranking and Deep Networks
Next article Differentiable Compositional Kernel Learning for Gaussian Processes
Raquel Urtasun is the Chief Scientist for Uber ATG and the Head of Uber ATG Toronto. She is also a Professor at the University of Toronto, a Canada Research Chair in Machine Learning and Computer Vision and a co-founder of the Vector Institute for AI. She is a recipient of an NSERC EWR Steacie Award, an NVIDIA Pioneers of AI Award, a Ministry of Education and Innovation Early Researcher Award, three Google Faculty Research Awards, an Amazon Faculty Research Award, a Connaught New Researcher Award, a Fallona Family Research Award and two Best Paper Runner up Prize awarded CVPR in 2013 and 2017. She was also named Chatelaine 2018 Woman of the year, and 2018 Toronto’s top influencers by Adweek magazine