Learning to Reweight Examples for Robust Deep Learning

    Abstract

    Deep neural networks have been shown to be very powerful modeling tools for many supervised learning tasks involving complex input patterns. However, they can also easily overfit to training set biases and label noises. In addition to various regularizers, example reweighting algorithms are popular solutions to these problems, but they require careful tuning of additional hyperparameters, such as example mining schedules and regularization hyperparameters. In contrast to past reweighting methods, which typically consist of functions of the cost value of each example, in this work we propose a novel meta-learning algorithm that learns to assign weights to training examples based on their gradient directions. To determine the example weights, our method performs a meta gradient descent step on the current mini-batch example weights (which are initialized from zero) to minimize the loss on a clean unbiased validation set. Our proposed method can be easily implemented on any type of deep network, does not require any additional hyperparameter tuning, and achieves impressive performance on class imbalance and corrupted label problems where only a small amount of clean validation data is available.

    Authors

    Mengye Ren, Wenyuan Zeng, Bin Yang, Raquel Urtasun

    Conference

    ICML 2018

    Full Paper

    ‘Learning to Reweight Examples for Robust Deep Learning’ (PDF)

    Uber ATG

    Comments
    Previous articleEvolving Multimodal Robot Behavior via Many Stepping Stones with the Combinatorial Multi-Objective Evolutionary Algorithm
    Next articleReviving and Improving Recurrent Back Propagation
    Mengye Ren
    Mengye Ren is a research scientist at Uber ATG Toronto. He is also a PhD student in the machine learning group of the Department of Computer Science at the University of Toronto. He studied Engineering Science in his undergrad at the University of Toronto. His research interests are machine learning, neural networks, and computer vision. He is originally from Shanghai, China.
    Avatar
    Wenyuan Zeng is currently a PhD student at the University of Toronto, supervised by Prof. Raquel Urtasun. His research interest mainly lies in deep learning, computer vision and decision making process. At the same time, he is also working full-time at Uber ATG Toronto to apply his research work to the development of self-driving cars, focusing on perception, prediction and planning. Before coming to University of Toronto, Wenyuan Zeng finished his Bachelor degree in Tsinghua University, China, majored in Mathematics and Physics.
    Avatar
    Bin Yang is a research scientist at Uber ATG Toronto. He's also a PhD student at University of Toronto, supervised by Prof. Raquel Urtasun. His research interest lies in computer vision and deep learning, with a focus on 3D perception in autonomous driving scenario.
    Raquel Urtasun
    Raquel Urtasun is the Chief Scientist for Uber ATG and the Head of Uber ATG Toronto. She is also a Professor at the University of Toronto, a Canada Research Chair in Machine Learning and Computer Vision and a co-founder of the Vector Institute for AI. She is a recipient of an NSERC EWR Steacie Award, an NVIDIA Pioneers of AI Award, a Ministry of Education and Innovation Early Researcher Award, three Google Faculty Research Awards, an Amazon Faculty Research Award, a Connaught New Researcher Award, a Fallona Family Research Award and two Best Paper Runner up Prize awarded CVPR in 2013 and 2017. She was also named Chatelaine 2018 Woman of the year, and 2018 Toronto’s top influencers by Adweek magazine