Skip to footer
Home Research Artificial Intelligence / Machine Learning End-to-end Learning of Multi-sensor 3D Tracking by Detection

End-to-end Learning of Multi-sensor 3D Tracking by Detection


In this paper we propose a novel approach to tracking by detection that can exploit both cameras as well as LIDAR data to produce very accurate 3D trajectories. Towards this goal, we formulate the problem as a linear program that can be solved exactly, and learn convolutional networks for detection as well as matching in an end-to-end manner. We evaluate our model in the challenging KITTI dataset and show very competitive results.


Davi Frossard, Raquel Urtasun


ICRA 2018

Full Paper

‘End-to-end Learning of Multi-sensor 3D Tracking by Detection’ (PDF)

Uber ATG

Previous article Pathwise Derivatives for Multivariate Distributions
Next article Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
Davi Frossard
I'm a research scientist at Uber ATG Toronto, developing cutting edge computer vision technology for self driving vehicles. Concurrently, I'm a PhD student at the University of Toronto doing research on computer vision under the supervision of prof. Raquel Urtasun. My research interests lie in the intersection of classic computational geometry, vision and machine learning. Previously, I did my Master's at the University of Toronto (also with prof. Urtasun) with the thesis on extracting visual vehicle attributes from a self driving platform and my Bachelor's degree at the Federal University of Espirito Santo (UFES) in Computer Engineering, writing my thesis on end-to-end learning of multiple object tracking.
Raquel Urtasun
Raquel Urtasun is the Chief Scientist for Uber ATG and the Head of Uber ATG Toronto. She is also a Professor at the University of Toronto, a Canada Research Chair in Machine Learning and Computer Vision and a co-founder of the Vector Institute for AI. She is a recipient of an NSERC EWR Steacie Award, an NVIDIA Pioneers of AI Award, a Ministry of Education and Innovation Early Researcher Award, three Google Faculty Research Awards, an Amazon Faculty Research Award, a Connaught New Researcher Award, a Fallona Family Research Award and two Best Paper Runner up Prize awarded CVPR in 2013 and 2017. She was also named Chatelaine 2018 Woman of the year, and 2018 Toronto’s top influencers by Adweek magazine