Deep learning currently provides the best representations of complex objects for a wide variety of tasks . But learning these representations is an expensive process that requires very large training samples and significant computing resources . Transferring representations commonly relies on the parameterized form of the features making up the representation, encoded by the computational graph of these features . In this paper, we propose to use a novel non-parametric metric between representations . It is based on a functional view of features, and takes into account certain invariances of representations, such as the permutation of their features, by relying on optimal transport . This distance is used as a regularization term promoting similarity between two representations . We show the relevance of this approach in two representation transfer settings, where the representation of a trained reference model is transferred to another one, for solving a new related task (inductive transfer learning), or for distilling knowledge to a simpler model (model compression). The approach is used in the model compression

Links: PDF - Abstract

Code :


Keywords : representations - model - representation - features - learning -

Leave a Reply

Your email address will not be published. Required fields are marked *


Enjoy this blog? Please spread the word :)