Skip to footer
Home Research Artificial Intelligence / Machine Learning SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability

0

Abstract

We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine transform (allowing comparison between different layers and networks) and fast to compute (allowing more comparisons to be calculated than with previous methods). We deploy this tool to measure the intrinsic dimensionality of layers, showing in some cases needless over-parameterization; to probe learning dynamics throughout training, finding that networks converge to final representations from the bottom up; to show where class-specific information in networks is formed; and to suggest new training regimes that simultaneously save computation and overfit less.

Authors

Maithra Raghu, Justin Gilmer, Jason Yosinski, Jascha Sohl-Dickstein

Conference

NIPS 2017

Full Paper

‘SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability’ (PDF)

Uber AI

Comments
Previous article Find Your Way by Observing the Sun and Other Semantic Cues
Next article Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning
Jason Yosinski
Jason Yosinski is a founding member of Uber AI Labs and there leads the Deep Collective research group. He is known for contributions to understanding neural network modeling, representations, and training. Prior to Uber, Jason worked on robotics at Caltech, co-founded two web companies, and started a robotics program in Los Angeles middle schools that now serves over 500 students. He completed his PhD working at the Cornell Creative Machines Lab, University of Montreal, JPL, and Google DeepMind. He is a recipient of the NASA Space Technology Research Fellowship, has co-authored over 50 papers and patents, and was VP of ML at Geometric Intelligence, which Uber acquired. His work has been profiled by NPR, the BBC, Wired, The Economist, Science, and the NY Times. In his free time, Jason enjoys cooking, reading, paragliding, and pretending he's an artist.