深度学习概述DL-(2).ppt

单击此处编辑母版标题样式,单击此处编辑母版文本样式,第二级,第三级,第四级,第五级,Institute of Computing Technology,Chinese Academy of Sciences,单击此处编辑母版标题样式,单击此处编辑母版文本样式,第二级,第三级,第四级,第五级,*,机器学习的基本任务,Class label,(Classification),Vector,(Estimation),dog,cat,horse,Object recognition,Super resolution,Low-resolution image,High-resolution image,1,源起,生物神经系统的启示,神经元之间通过突触,(synapse),连接,层级感受野，学习使突触连接增强或变弱甚至消失,Hubel,D.H.&Wiesel,T.N.(1962),2,第一代神经网络,Frank Rosenblatt(1957),The Perceptron-a perceiving and recognizing automaton.Report 85-460-1,Cornell Aeronautical Laboratory.,3,Error Backpropagation,W is the parameter of the network;J is the objective function,Feedforward operation,Back error propagation,David E.,Rumelhart,Geoffrey E.,Hinton,and,Ronald,J.Williams.,(Oct.1986).Learning representations by back-propagating errors.,Nature,323,(6088):533536,Output layer,Hidden layers,Input layer,Target values,2,nd,Generation Neural Networks,理论上多层好,两层权重即可逼近任何连续函数映射,遗憾的是，训练困难,It requires labeled training data,Almost all data is unlabeled.,The learning time does not scale well,It is very slow in networks with multiple hidden layers.,It can get stuck in poor local optima,These are often quite good,but for deep nets they are far from optimal.,7,1990-2006,更流行,Specific methods for specific tasks,Hand-crafted features(SIFT,LBP,HOG),ML methods,SVM,Kernel tricks,Boosting,AdaBoost,kNN,Decision tree,Kruger et al.TPAMI13,8,A Breakthrough Back to 2006,2006,年，通过分层的、无监督预训练，终于获得了训练深层网络结构的能力,9,A Breakthrough Back to 2006,Hinton,G.E.,Osindero,S.and Teh,Y.,A fast learning algorithm for,deep belief nets,.Neural Computation 18:1527-1554,2006,Hinton,G.E.and Salakhutdinov,R.R.(2006),Reducing the dimensionality,of data with neural networks.Science,Vol.313.no.5786,pp.504-507,28 July 2006,Yoshua Bengio,Pascal Lamblin,Dan Popovici and Hugo Larochelle,Greedy,Layer-Wise Training of Deep Networks,Advances in Neural Information Processing Systems 19(NIPS 2006),MarcAurelio Ranzato,Christopher Poultney,Sumit Chopra and,Yann LeCun.,Efficient Learning of Sparse Representations,with an Energy-Based Model,Advances in Neural Information Processing Systems(NIPS 2006),10,其实是有例外的,CNN,卷积神经网络,CNN,K.,Fukushima,“Neocognitron:A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,”,Biological,Cybernetics,vol.36,pp.193202,1980,Y.LeCun,B.Boser,J.S.Denker,D.Henderson,R.E.Howard,W.Hubbard,and L.D.Jackel,“Backpropagation applied to handwritten zip code recognition,”Neural Computation,vol.1,no.4,pp.541551,1989,Y.Le Cun,L.Bottou,Y.Bengio,and P.Haffner,“Gradient-based learning applied to document recognition,”Proceedings of the IEEE,vol.86,no.11,pp.22782324,1998,11,其实是有例外的,CNN,Neocognitron 1980,K.,Fukushima,“Neocognitron:A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,”,Biological,Cybernetics,vol.36,pp.193202,1980,Local Connection,12,例外：,CNN,用于数字识别,13,例外：,CNN,用于目标检测与识别,14,而且，东风同样重要,大数据,大数据,大数据,语音图像视频,计算能力,并行计算平台,GPU,大量部署,开放的社区,开源，开放数据,15,语音识别,(2011),1986,2006,DBN,Science,Speech,2011,BP,16,2012,年计算机视觉的巨大进步,ImageNet,物体分类任务上,物体分类任务：,1000,类，,1,431,167,幅图像,1986,2006,DBN,Science,Speech,2011,2012,Rank,Name,Error rates(TOP5),Description,1,U.Toronto,0.153,Deep learning,2,U.Tokyo,0.261,Hand-crafted,features and learning models.,Bottleneck.,3,U.,Oxford,0.270,4,Xerox/INRIA,0.271,BP,17,ImageNet with Deep CNN,方法：大规模,CNN,网络,A.Krizhevsky,L.Sutskever,and G.E.Hinton,“ImageNet Classification with Deep Convolutional Neural Networks,”NIPS,2012.,18,ImageNet with Deep CNN,方法：大规模,CNN,网络,650K,神经元,60M,参数,Trained with BP on GPU,使用了各种技巧,+,dropout,ReLU,Data augment,contrast normalization,.,被,Google,收编,(Jan 2013),Google+Photo Tagging(2013.5),A.Krizhevsky,L.Sutskever,and G.E.Hinton,“ImageNet Classification with Deep Convolutional Neural Networks,”NIPS,2012.,19,ImageNet,物体分类,(2013),1000,类，,1,431,167,幅图像，,Top 5,错误率,1986,2006,DBN,Science,Speech,2011,2012,Rank,Name,Error rates(TOP5),Description,1,NYU,0.11197,Deep learning,2,NUS,0.12535,Deep learning,3,Oxford,0.13555,Deep learning,2013,BP,20,MIT Tech Review,坐不住了,21,ImageNet,物体分类,(2014),1000,类，,1,431,167,幅图像，,Top 5,错误率,1986,2006,DBN,Science,Speech,2011,2012,Rank,Name,Error rates(TOP5),Description,1,Google,006656,Deep learning,2,Oxford,0.07325,Deep learning,3,MSRA,0.08062,Deep learning,2013,2014,BP,22,ImageNet,物体分类,(2014),GoogLeNet CVPR2015,22,个卷积层,Szegedy C,Liu W,Jia Y,et al.Going deeper with convolutions.CVPR 2015,1986,2006,DBN,Science,Speech,2011,2012,BP,2014,2013,23,ImageNet,物体分类,(2010-2014),ImageNet Top 5 Error Rate,上的持续进步,1986,2006,DBN,Science,Speech,2011,2012,BP,2014,2013,24,ImageNet,物体检测任务,200,类，,456,567,幅图像，检测率,传统方法,SIFT+BOW+SPM,深度方法,R-CNN+GoogLeNet,1986,2006,DBN,Science,Speech,2011,2012,BP,2014,2013,25,物体分割,/,语义标注进步迅速,Jonathan Long,Evan Shelhamer,Trevor Darrell.,Fully Convolutional Networks for Semantic Segmentation.CVPR2015,26,DL,有多热,Deep Learning for Vision,602,篇文章中，仅标题中出现,Deep,的就有,87,篇，出现,Convolution,的,47,篇，出现,Neural,的,40,篇，出现,Network,的,51,篇，,Recurrent,7,篇,Going deeper,，优化，无监督、自主学习,Fully Convolutional Network,（,for segmentation,等）,Vision and Language(for,看图说话,Google,Fei-fei,Microsoft,UCB),RNN with LSTM,（,for,时序处理）,Deep Learning for X(detection,metric learning,attribute,hash,),27,计算机视觉的重大进步,Vision and Language(Google,Microsoft,UCB),看图说话：,Minsky 60,年前布置的作业,Show and Tell:A Neural Image Caption Generator(a work from Google),From Captions to Visual Concepts and Back(a work from Microsoft),Long-term Recurrent Convolutional Networks for Visual Recognition and Description(a work from UTA/UML/UCB),28,人脸识别上的进步,正确率,95.17,%,D.Chen,X.Cao,F.Wen,J.Sun,CVPR13,正确率,97.35%,Y.Taigman,M.Yang,M.Ranzato,L.Wolf,CVPR14,正确率,99.47%,Y.Sun,X.Wang,and X.Tang,CVPR14,正确率,99.63,%,F.Schroff,D.Kalenichenko,and J.Philbin,CVPR15,1986,2006,DBN,Science,Speech,2011,2012,Face,2014,2015,BP,在,LFW,上，过去,2,年错误率从,5%,下降到,0.5%,(,错,300,对,错,30,对,),29,人脸识别上的进步,Labeled Face in the Wild(LFW),非限定条件下的人脸识别,数据来源于因特网,国外名人，,Yahoo,新闻,广为人知的测试模式,训练集：,无限制,验证任务测试集,共,6000,图像对,Huang G B,Ramesh M,Berg T,et al.Labeled faces in the wild:A database for studying face recognition in unconstrained environments.Technical Report,University of Massachusetts,Amherst,2007.,30,人脸识别上的进步,2014:DeepFace 1(Facebook),大数据：,4K,人，,4.4M,图像,1 Taigman Y,Yang M,Ranzato M A,et al.Deepface:Closing the gap to human-level performance in face verification.CVPR,2014.,2 Sun Y,Wang X,Tang X.Deeply learned face representations are sparse,selective,and robust.arXiv preprint,2014.,31,人脸识别上的进步,香港中文大学,DeepID2+,在,25,个人脸,Patch,上分别训练,CNN,（,4,个卷积层，,4,个全连接层，,4,个,verification,损失信号和,1,个,identification,损失信号）,训练数据：,10K,人，,202K,名人图像,Y.Sun,X.Wang,and X.Tang,CVPR14,人脸识别上的进步,Google,最新的,FaceNet,深层网络（,22,层）,+,海量数据,(800,万人，,2,亿张图像）,+Triplet Loss(,不需要额外占用显存,),F.Schroff,D.Kalenichenko,and J.Philbin,CVPR15,提纲,深度学习,(DL),及其应用前沿,DL,在,CV,领域应用的启示,关键算法介绍,BP,算法,Auto-Encoder,CNN,CNN,主要变种,关于,DL,的思考与讨论,34,DL,之前的视觉处理方法,分步处理背后的哲学,分而治之,Divide and Conquer,Knowledge-driven,Hand-crafted feature,I think it should be solved by methods like,35,DL,及其之后的视觉处理方法,36,DL,及其之后的视觉处理方法,学习到接近期望的底层、中层和高层特征,37,DL,之前的视觉处理方法,任务,人工设计,F,（部分学习,F,）,领域知识：分步处理,滤波器，局部特征,(SIFT),，,BoW,，直方图，,Max/Sum,汇聚，,判别分析，,Kernel,技巧，分段线性，流形学习，测度学习,类标签,(,分类问题,),向量,(,回归,/,估计,),预处理,特征设计,特征降维,分类,/,回归,38,DL,时代的视觉处理方法,任务人工设计,F,（部分学习,F,）,End-to-end,地学习,F,（全步骤学习）,Representation learning,Feature learning,Nonlinear transform learning,离散类标签,(,分类问题,),连续向量,(,回归,/,估计,),Credit to Dr.Xiaogang Wang,39,DL,时代的视觉处理方法,Collect data,Preprocessing 1,Feature design,Classifier,Evaluation,Preprocessing 2,Collect data,Feature transformFeature transform,Classifier,Deep neural network,Evaluation,vs.,Credit to Dr.Xiaogang Wang,40,DL,时代的视觉处理方法,方法论上的变化,从分治,协同（,joint,）,多步骤,end-to-end learning,更广义的,检测与识别,分割与识别,41,提纲,深度学习,(DL),及其应用前沿,DL,在,CV,领域应用的启示,关键算法介绍,Perceptron,算法,BP,算法,Auto-Encoder,CNN,及其主要变种,关于,DL,的思考与讨论,42,Perceptron,Frank Rosenblatt(1957),The Perceptron-a perceiving and recognizing automaton.Report 85-460-1,Cornell Aeronautical Laboratory.,43,Perceptron,算法,F.Rosenblatt.The perceptron:A probabilistic model for informationstorage and organization in the brain.,Psychological Review,65:386-408,1958,44,Perceptron,算法,45,前馈神经网络的,BP,学习算法,David E.,Rumelhart,Geoffrey E.,Hinton,and Ronald J.,Williams,.(Oct.1986).Learning representations by back-propagating errors.,Nature,323,(6088):533536,（单独,slides,）,卷积神经网络及其变种,（单独,slides,）,47,提纲,深度学习,(DL),及其应用前沿,DL,在,CV,领域应用的启示,关键算法介绍,Perceptron,及学习算法,MLP,及其,BP,算法,Auto-Encoder,CNN,及其主要变种,关于DL,的思考与讨论,48,关于,DL,的更多讨论,DL,带来观念的变革,DL,是类脑信息处理方法吗？,DL,有理论吗？,DL,不能做什么？,数据驱动的学习不再需要领域知识？,工业界抢了学术界的饭碗？,CV,研究者沦为,ML,研究者的实验员？,DL,未来工作？,49,DL,带来观念的变革,人工领域知识驱动,数据驱动的学习思想,小数据,控制模型复杂度避免过拟合,大数据,提高模型复杂度避免欠拟合,“大数据,+,简单模型”是错误的！,维数灾难,(,降维,),高维有益,(,升维,),分步、分治思想,协同学习,(joint learning),思想,End-to-end,的全过程学习,软硬件更优的协同,50,DL,是类脑信息处理方法吗？,DL,受到脑信息处理方式启发,分层逐级抽象,初级视觉神经元的“类,Gabor,小波编码”,并不“类脑”,本质上，脑的计算“机制”尚不清晰,脑的连接更多样、更复杂,Top-down,，反馈机制,学习过程未必需要大量数据,先天,生物进化的结果（大数据长期训练）,后天学习,更多演绎推理，迁移学习,51,DL,有理论吗？,DL,理论匮乏,收敛性，,bound,局部极值，初值很重要,复杂度理论,但不完全是,black box,与传统“分步”做法的关系,比,Kernel,更“显式”,层级可视化提供了很多线索,逐层抽象,or,分层“非线性”？,52,DL,不能做什么？,用做“特征学习”或“非线性变换”最成功,学到的特征具有良好的通用性,传统分类器或回归似乎还可用,非常倚重大数据，小数据深度学习不可靠,需要引入领域知识，深度模型的迁移学习,难以演绎推理,DL,是归纳学习，难以举一反三，更难无师自通,在一些简单问题上未必需要深度学习,人脸识别的例子,目前的,DL,不学习“自身结构”,调试经验很重要,53,数据驱动的学习不需要领域知识？,大数据驱动确实减少了对领域知识的依赖,CNN,在,CV,领域的成功，本身就说明了领域知识的重要性,卷积操作，,Pooling,操作,小数据条件下，领域知识尤其重要,Data is king,and DL is queen?,54,工业界抢了学术界的饭碗？,工业界看不起学术界？,工业界重视大数据收集和并行实现,学术界重视理论和新的模型,CV,学术界应该更,smart,更前瞻,新的模型设计,网络结构学习,优化方法,训练加速,更,Smart,的数据收集,大而脏乱差数据的高效利用,55,CV,研究者沦为,ML,研究者的实验员？,有这个危险,CV,本身缺少理论体系,“分步”法主宰,CV,太多年,CV,仍有机会,实际上,ML,也从,CV,获益良多,基于学习的,CV,理论？,CV,研究者应该与,ML,有更多互动,“几何”和“结构”的可学习性？,56,DL,领域的未来工作,DL,理论,网络结构本身的学习,小数据条件下的,DL,领域知识的嵌入,带反馈的深度网络,大而脏乱差数据条件下的,DL,深度模型的迁移与适应,面向视频分析的,DL,模型,“非线性”的更多来源,新的优化和训练算法,57,总结和警告！,神经网络兴衰史的教训,是复兴不是创新,历史经常重演,相比,CV,等应用领域对,DL,的狂热，,ML,领域很冷静,他强由他强，清风拂山冈；,他横由他横，明月照大江。,建议,要会,DL,，但不要只会,DL,经验知识驱动,数据驱动,混合驱动,58,