Document

Download Report

Transcript Document

Can Machines Transfer Knowledge from Task to Task?

Isabelle Guyon Clopinet, California

http://clopinet.com/ul

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

CREDITS

Data donors: Handwriting recognition (AVICENNA) - Reza Farrahi Moghaddam, Mathias Adankon, Kostyantyn Filonenko, Robert Wisnovsky, and Mohamed Chériet (Ecole de technologie supérieure de Montréal, Quebec) contributed the dataset of Arabic manuscripts. The toy example (ULE) is the MNIST handwritten digit database made available by Yann LeCun and Corinna Costes.

Object recognition (RITA) -- Antonio Torralba, Rob Fergus, and William T. Freeman, collected and made available publicly the 80 million tiny image dataset. Vinod Nair and Geoffrey Hinton collected and made available publicly the CIFAR datasets. See the techreport Learning Multiple Layers of Features from Tiny Images, by Alex Krizhevsky, 2009, for details.

Human action recognition (HARRY) -- Ivan Laptev and Barbara Caputo collected and made publicly available the KTH human action recognition datasets. Marcin Marszałek, Ivan Laptev and Cordelia Schmid collected and made publicly available the Hollywood 2 dataset of human actions and scenes.

Text processing (TERRY) -- David Lewis formatted and made publicly available the RCV1-v2 Text Categorization Test Collection.

Ecology (SYLVESTER) -- Jock A. Blackard, Denis J. Dean, and Charles W. Anderson of the US Forest Service, USA, collected and made available the (Forest cover type) dataset.

Web platform: Server made available by Prof. Joachim Buhmann, ETH Zurich, Switzerland. Computer admin.: Thomas Fuchs, ETH Zurich. Webmaster: Olivier Guyon, MisterP.net, France. Platform: Causality Wokbench.

Co-orgnizers: • David W. Aha, Naval Research Laboratory, USA.

• Gideon Dror, Academic College of Tel-Aviv Yaffo, Israel.

• Vincent Lemaire, Orange Research Labs, France.

• Graham Taylor, NYU, New-York. USA.

• Gavin Cawley, University of east Anglia, UK.

• Danny Silver, Acadiau University, Canada.

• Vassilis Athitsos, UT Arlington, Texas., USA.

Protocol review and advising: • Olivier Chapelle, Yahoo!, California, USA.

• Gerard Rinkus, Brandeis University, USA.

• Urs Mueller, Net-Scale Technilogies, USA.

• Yoshua Bengio, Universite de Montreal, Canada.

• David Grangier, NEC Labs, USA.

• Andrew Ng, Stanford Univ., Palo Alto, California, USA.

• Yann LeCun, NYU. New-York, USA.

• Richard Bowden, University of Surrey, UK.

• Philippe Dreuw, Aachen University, Germany.

• Ivan Laptev, INRIA, France.

• Jitendra Malik, UC Berkeley, USA.

• Greg Mori, Simon Fraser University, Canada. • Christian Vogler, ILSP, Athens, Greece Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

What is the problem?

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Can learning about...

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

help us learn about…

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Can learning about…

publicly available data

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

help us learn about…

personal data

Philip and Thomas Anna Solene Anna, Thomas and GM Omar, Thomas Philip Philip Martin Bernhard Philip Thomas

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Transfer learning

Common data representation

Philip and Thomas Anna Solene Anna, Thomas and GM Omar, Thomas Philip Philip Martin Bernhard Philip Thomas

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

How?

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Vocabulary

Source task labels Target task labels

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Vocabulary

Source task labels Target task labels

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Vocabulary

Source task labels Labels available?

Tasks the same?

Target task labels Domains the same?

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Transfer Learning

Labels available in target domain

Taxonomy of transfer learning

No labels in source domain

Self-taught TL Inductive TL

Labels available in source domain

Multi-task TL

Same source and target task

Transductive TL

Labels avail. ONLY in source domain

Semi-supervised TL

Different source and target tasks

Cross-task TL

No labels in both source and target domains

Unsupervised TL

Unsupervised and Transfer Learning Challenge Adapted from: A survey on transfer learning, Pan-Yang, 2010.

http://clopinet.com/ul

Transfer Learning

Labels available in target domain

Taxonomy of transfer learning

No labels in source domain

Self-taught TL Inductive TL

Labels available in source domain

Multi-task TL

Same source and target task

Transductive TL

Labels avail. ONLY in source domain

Semi-supervised TL

Different source and target tasks

Cross-task TL

No labels in both source and target domains

Unsupervised TL

Unsupervised and Transfer Learning Challenge Adapted from: A survey on transfer learning, Pan-Yang, 2010.

http://clopinet.com/ul

Unsupervised transfer learning

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

What can you do with NO labels?

• No learning at all: – Normalization of examples or features – Construction of features (e.g. products) – Generic data transformations (e.g. taking the log, Fourier transform, smoothing, etc.) • Unsupervised learning: – Manifold learning to reduce dimension (and/or orthogonalize features) – Sparse coding to expand dimension – Clustering to construct features – Generative models and latent variable models

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

1) Source domain

Unsupervised transfer learning

P R

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Unsupervised transfer learning

P

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Unsupervised transfer learning

P

2) Target domain

P

Unsupervised and Transfer Learning Challenge

C

John

Task labels

http://clopinet.com/ul

Unsupervised transfer learning

Target domain

P

Unsupervised and Transfer Learning Challenge

C

Emily 20

http://clopinet.com/ul

Manifold learning

• PCA • ICA • Kernel PCA • Kohonen maps • Auto-encoders • MDS, Isomap, LLE, Laplacian Eigenmaps • Regularized principal manifolds

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Deep Learning

Greedy layer-wise unsupervised pre-training of multi-layer neural networks and Bayesian networks, including: • Deep Belief Networks (stacks of Restricted Boltzmann machines) • Stacks of auto-encoders

reconstructor

Unsupervised and Transfer Learning Challenge

preprocessor

http://clopinet.com/ul

Clustering

• K-means

and variants w. cluster overlap (Gaussian mixtures, fuzzy C-means)

• Hierarchical clustering • Graph partitioning • Spectral clustering

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Example: K-means

•

Start with random cluster centers.

•

Iterate:

Assign the examples to their closest center to form clusters.

Re-compute the centers by averaging the cluster members.

•

Create features, e.g.

f k = exp – g

||x-x

Unsupervised and Transfer Learning Challenge

Clusters of ULE valid after 5 it.

http://clopinet.com/ul

Results on ULE: do better!

ALC=0.79

ALC=0.84

log2(num. tr. ex.) Raw data: 784 features log2(num. tr. ex.) K-means: 20 features Current best: AUC=1, ALC=0.96

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Unsupervised learning (resources)

• • • • • •

Unsupervised Learning

. Z. Ghahramani. http://www.gatsby.ucl.ac.uk/~zoubin/course04/ul.pdf

Nonlinear dimensionality reduction

. http://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction

Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering.

Y. Bengio et al. http://books.nips.cc/papers/files/nips16/NIPS2003_AA23.pdf

Data Clustering

: A Review. Jain et al.

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.2720

Why Does Unsupervised Pre-training Help DL?

D. Erhan et al. http://jmlr.csail.mit.edu/papers/volume11/erhan10a/erhan10a.pdf

Efficient sparse coding algorithms.

H. Lee et al. http://www.eecs.umich.edu/~honglak/nips06 26

http://clopinet.com/ul

Transfer Learning

Labels available in target domain

Taxonomy of transfer learning

No labels in source domain

Self-taught TL Inductive TL

Labels available in source domain

Multi-task TL

Same source and target task

Transductive TL

Labels avail. ONLY in source domain

Semi-supervised TL

Different source and target tasks

Cross-task TL

No labels in both source and target domains

Unsupervised TL

Unsupervised and Transfer Learning Challenge Adapted from: A survey on transfer learning, Pan-Yang, 2010.

http://clopinet.com/ul

Cross-task transfer learning

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

How can you do it?

• Data representation learning: – Deep neural networks – Deep belief networks (re-use the internal representation created by the hidden units and/or output units) • Similarity or kernel learning: – Siamese neural networks – Graph-theoretic methods

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

1) Source domain

Data representation learning

P C

Sea

Source task labels

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Data representation learning

P

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Data representation learning

P

2) Target domain

P

Unsupervised and Transfer Learning Challenge

C

John

Target task labels

http://clopinet.com/ul

Data representation learning

Target domain

P

Unsupervised and Transfer Learning Challenge

C

Emily 33

http://clopinet.com/ul

Kernel learning

P

Source domain

P

Unsupervised and Transfer Learning Challenge

S

same or different

Source task labels

http://clopinet.com/ul

Kernel learning

P

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Kernel learning

P

2) Target domain

P

Unsupervised and Transfer Learning Challenge

C

John

Target task labels

http://clopinet.com/ul

Kernel learning

Target domain

P

Unsupervised and Transfer Learning Challenge

C

Emily 37

http://clopinet.com/ul

Cool results in cross-task transfer learning

Genuine or not Source task Target tasks

NLP (almost) from scratch. Collobert et al. 2011, submitted to JMLR Unsupervised and Transfer Learning Challenge

pos=Part-Of-Speech tagging chunk=Chunking ner=Named Entity Recognition srl=Semantic Role Labeling 38

http://clopinet.com/ul

Cross-task transfer (resources)

• •

A Survey on Transfer Learning

. Pan and Yang. http://www1.i2r.a star.edu.sg/~jspan/publications/TLsurvey_0822.pdf

Distance metric learning: A comprehensive survey

. Yang-Jin.

• • • http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.91.47

Signature Verification using a "Siamese" Time Delay Neural Network

. Bromley et al. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.28.4792

Learning the kernel matrix with semi-definite programming

, Lanckriet et al. http://jmlr.csail.mit.edu/papers/volume5/lanckriet04a/lanckriet04 a.pdf

NLP (almost) from scratch. http://clopinet.com/ul

http://leon.bottou.org/morefiles/nlp.pdf

Transfer Learning

Labels available in target domain

Taxonomy of transfer learning

No labels in source domain

Self-taught TL Inductive TL

Labels available in source domain

Multi-task TL

Same source and target task

Transductive TL

Labels avail. ONLY in source domain

Semi-supervised TL

Different source and target tasks

Cross-task TL

No labels in both source and target domains

Unsupervised TL

Unsupervised and Transfer Learning Challenge Adapted from: A survey on transfer learning, Pan-Yang, 2010.

http://clopinet.com/ul

Multi-task learning

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Multi-task learning

Source domain Target domain

P

Unsupervised and Transfer Learning Challenge

C

Sea John

Source task labels Target task labels

http://clopinet.com/ul

Multi-task learning

Target domain

P C

Emily

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Cool results in multi-task learning

One-Shot Learning with a Hierarchical Nonparametric Bayesian Model, Salakhutdinov-Tenenbaum-Torralba, 2010 Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Transfer Learning

Labels available in target domain

Taxonomy of transfer learning

No labels in source domain

Self-taught TL Inductive TL

Labels available in source domain

Multi-task TL

Same source and target task

Transductive TL

Labels avail. ONLY in source domain

Semi-supervised TL

Different source and target tasks

Cross-task TL

No labels in both source and target domains

Unsupervised TL

Unsupervised and Transfer Learning Challenge Adapted from: A survey on transfer learning, Pan-Yang, 2010.

http://clopinet.com/ul

Self-taught learning

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Self-taught learning

Source domain Target domain

P

Unsupervised and Transfer Learning Challenge

C

John

Target task labels

http://clopinet.com/ul

Self-taught learning

Target domain

P C

Emily

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Source task

Cool results in self-taught learning

Target task

Unsupervised Semi-supervised Multi-task Self-taught

Self-taught learning. R. Raina et al. 2007 Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Inductive transfer learning (resources)

• • • • •

Multitask learning

. R. Caruana. http://www.cs.cornell.edu/~caruana/mlj97.pdf

Learning deep architectures for AI

. Y. Bengio. http://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf

Transfer Learning Techniques for Deep Neural Nets

. S. M. Gutstein thesis. http://robust.cs.utep.edu/~gutstein/sg_home_files/thesis.pdf

One-Shot Learning with a Hierarchical Nonparametric Bayesian Model.

R. Salakhutdinov et al. http://dspace.mit.edu/bitstream/handle/1721.1/60025/ MIT-CSAIL-TR-2010-052.pdf?sequence=1

Self-taught learning.

R. Raina et al. 50 http://www.stanford.edu/~rajatr/papers/icml07_SelfTau

http://clopinet.com/ul

Dec 2010-April 2011 http://clopinet.com/ul Development data Validation data Challenge data Competitors Source task labels Data represen tations Validation target task labels Challenge target task labels

•

Goal: Learning data representations or kernels.

•

Phase 1:

Unsupervised learning (until Feb. 28) •

Phase 2:

Cross-task transfer learning (from Mar. 1) •

Prizes:

$6000 + free • registrations + travel awards

Dissemination:

Workshops at ICML and IJCNN; proc. in JMLR W&CP.

Evaluators

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

July 2011, ICML - Dec 2011, NIPS http://clopinet.com/tl Development Data (source + target data) Competitors All task labels Multi-task learning setting:

- Synthetic, Real-world - Supervised learning - Binary classification problems.

- 5-10 secondary tasks, 1 primary -

Impoverished primary task data in development set

Diversity of tasks with varying degree of relatedness to primary task

Validation data (target only) Challenge data (target only) Predic tions Target task validation labels Target task challenge labels

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Challenge

June 2011-June. 2012 http://clopinet.com/gs (in preparation) STEP 1: Develop a “generic” sign language recognition system that can learn new signs with a few examples.

STEP 2: At conference: teach the system new signs.

STEP 3: Live evaluation in front of audience.

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul

Conclusion

• Transfer learning algorithms offer solutions to problems in which – a lot of training samples are available for a

source task,

– fewer training samples are available for a similar but different

target task.

• We stated a program of challenges featuring problems in which transfer learning is applicable.

Unsupervised and Transfer Learning Challenge

http://clopinet.com/ul